0:0018:19

Observability In Data Ingestion

Nancy Chauhan 18:19 2022

Your data pipeline will never be error free, so build it to tell you why

One incorrectly processed city in a payments dataset is enough to send analytics for every city the wrong way, and at large volumes some bad data is a certainty. Software engineer Nancy Chauhan, who has spent two years building solutions for software reliability, explains why ingestion pipelines need observability. What should a team measure, and where should it look, to catch failures before the business acts on them?

Learn the questions and core metrics that make a batch or streaming ingestion pipeline observable

Watch on YouTube

In short

Software engineer Nancy Chauhan explains what data ingestion is and why pipelines handling large batch and streaming volumes need observability rather than basic monitoring. She argues that errors are inevitable at scale, so teams should ask questions about freshness, volume, distribution, lineage and schema, then track record counts, efficiency and error messages in a single dashboard, illustrated with a Stackdriver example on Google Cloud.

“So observability simply answers that why X is broken.”
“So we need to ensure if you have incorrect data that should not affect your correct data.”

What you will learn

  • Explain how data ingestion collects, remaps and stores data from websites, apps, CRMs and other sources.
  • Distinguish observability from basic monitoring and recognise why distributed systems need it.
  • Ask the freshness, volume, distribution, lineage and schema questions before designing a data observability solution.
  • Track records at the source against records ingested to measure pipeline efficiency and spot dropped data.
  • Use a single dashboard of error rates and error messages, and tools like Stackdriver, to monitor streaming sources.

Key takeaways from the speaker

  • Data ingestion
  • Data engineering
  • CDAP
  • Data monitoring

Details

Level
Intermediate
Format
Talk
Technical depth
Introductory technical
For
Early career, Mid-career
Length
18:19
Published
July 2022

Questions this talk answers

What is data ingestion?

It is the process of pulling data from many sources, such as websites, mobile apps and CRMs via APIs, SDKs, server integrations or manual imports, remapping it into a required format and storing it in a target database for analytics.

How is observability different from monitoring?

Monitoring CPU, memory and network is usually enough for simple systems, but observability suits unpredictable distributed systems because it lets you ask why something is broken as issues arise.

Why does data ingestion need observability?

Large data volumes are never 100% error free, and delayed or incorrectly processed data can lead to wrong business decisions and lost revenue. Observability also gives clients transparency about why their data succeeded or failed.

What should a team ask before building a data observability solution?

Ask about freshness (is data on time), volume (did all data arrive), distribution (where was it delivered), lineage (upstream sources and downstream consumers) and schema (is it in the correct format).

Which metrics help observe an ingestion pipeline?

Track data volume per pipeline run, total records at the source versus after ingestion to calculate efficiency, error rates and error messages, ideally all in one dashboard with alerting.

About this talk

Data has become an essential part of the tech stack of modern organizations. Everyone deals with ingestion, transformation and storage of huge volumes of data which are critical to business analytics operations.

This talk is about how we can observe the ingestion of large datasets and streaming sources, various transformation operations and their storage and why is it important. Observabilty helps to determine completeness, accuracy, efficiency and consistency for any data uploaded through either batch or streaming sources. As well as helps to identify failures. It provides necessary details to customers about ingestion success or failure and enables them to take corrective action. It also reduces the time invested by the tech team to investigate the errors.

Topics

Related talks

Browse the library