Your data pipeline will never be error free, so build it to tell you why
One incorrectly processed city in a payments dataset is enough to send analytics for every city the wrong way, and at large volumes some bad data is a certainty. Software engineer Nancy Chauhan, who has spent two years building solutions for software reliability, explains why ingestion pipelines need observability. What should a team measure, and where should it look, to catch failures before the business acts on them?
Learn the questions and core metrics that make a batch or streaming ingestion pipeline observable
Software engineer Nancy Chauhan explains what data ingestion is and why pipelines handling large batch and streaming volumes need observability rather than basic monitoring. She argues that errors are inevitable at scale, so teams should ask questions about freshness, volume, distribution, lineage and schema, then track record counts, efficiency and error messages in a single dashboard, illustrated with a Stackdriver example on Google Cloud.
“So observability simply answers that why X is broken.”
“So we need to ensure if you have incorrect data that should not affect your correct data.”
What you will learn
Explain how data ingestion collects, remaps and stores data from websites, apps, CRMs and other sources.
Distinguish observability from basic monitoring and recognise why distributed systems need it.
Ask the freshness, volume, distribution, lineage and schema questions before designing a data observability solution.
Track records at the source against records ingested to measure pipeline efficiency and spot dropped data.
Use a single dashboard of error rates and error messages, and tools like Stackdriver, to monitor streaming sources.
Key takeaways from the speaker
Data ingestion
Data engineering
CDAP
Data monitoring
Details
Level
Intermediate
Format
Talk
Technical depth
Introductory technical
For
Early career, Mid-career
Length
18:19
Published
July 2022
Questions this talk answers
What is data ingestion?
It is the process of pulling data from many sources, such as websites, mobile apps and CRMs via APIs, SDKs, server integrations or manual imports, remapping it into a required format and storing it in a target database for analytics.
How is observability different from monitoring?
Monitoring CPU, memory and network is usually enough for simple systems, but observability suits unpredictable distributed systems because it lets you ask why something is broken as issues arise.
Why does data ingestion need observability?
Large data volumes are never 100% error free, and delayed or incorrectly processed data can lead to wrong business decisions and lost revenue. Observability also gives clients transparency about why their data succeeded or failed.
What should a team ask before building a data observability solution?
Ask about freshness (is data on time), volume (did all data arrive), distribution (where was it delivered), lineage (upstream sources and downstream consumers) and schema (is it in the correct format).
Which metrics help observe an ingestion pipeline?
Track data volume per pipeline run, total records at the source versus after ingestion to calculate efficiency, error rates and error messages, ideally all in one dashboard with alerting.
About this talk
Data has become an essential part of the tech stack of modern organizations. Everyone deals with ingestion, transformation and storage of huge volumes of data which are critical to business analytics operations.
This talk is about how we can observe the ingestion of large datasets and streaming sources, various transformation operations and their storage and why is it important. Observabilty helps to determine completeness, accuracy, efficiency and consistency for any data uploaded through either batch or streaming sources. As well as helps to identify failures. It provides necessary details to customers about ingestion success or failure and enables them to take corrective action. It also reduces the time invested by the tech team to investigate the errors.
How one corrupted city skews every decision (68 s)
One dashboard for every pipeline failure (66 s)
Hey everyone, this is Nancy Chauhan. Welcome to my session. It's about observability in data ingestion. I'm so excited to have you here. I'll just start my screen sharing. Awesome. So today we're going to talk about observability in data ingestion. Now, before we start, why should you attend this talk? Well, data has become an essential part of almost all the organization and almost everyone deals with ingestion, transformation, storage of huge volumes of data, which is very critical for all the business analytics operation. So this talk is about data ingestion. And when we have data ingestion, we want some control system by which we can determine the accuracy, efficiency and consistency of our system so that we can catch, we can identify the failures. And that's why we need observability. So observability, it will help you to determine the completeness, accuracy, efficiency and consistency for any data which is uploaded through either batch or streaming sources.
Speaker 1: And it also reduces the time invested by the tech team to investigate the errors because you will always have the errors in front of you. Speaker 1: A little bit about me. I am Nancy. I am a developer and tech writer. I'm currently contributing at Gitpod. Gitpod is it's a remote, Speaker 1: It's an open source, open source remote development software tool and remote is the future. And that's how we are here doing the conversation, doing the conference remotely.
Speaker 1: Previously, I worked at Growfers and Zeodap and I was developing solutions for software reliability. Last year, I did the talk on monitoring and this year I'm doing the talk on observability. Speaker 1: I love open source. I keep contributing and I really love talking about tech through blogs. So you can check out my blogs at nancychohan.in.
Speaker 1: Okay, so let's get started. What is data ingestion? Well, data ingestion is the process of connecting wide variety of data structures into where it needs to be given into required format and quality. And this may be a storage medium or application for further processing. So basically, what is data ingestion? Speaker 1: We are pulling a lot of data from different sources and then we are going to map it and then we are going to store it into the target database. So basically, we can say the focus of data ingestion is to get data into the system that requires data in a particular format for other operational Speaker 1: users, which is later basically used for mainly for data business analytics operation. Let's just understand this with a very nice illustration, which has been done by me.
Speaker 1: So as you can see, there's a user and we can collect various information from various sources, which can be website, your mobile applications. You keep surfing mobile phone. It can be CRMs. And data is collected through various methods. Speaker 1: Like like we have APIs or we can we can also have even trackers. We have for example, we have JavaScript tags and SDKs or maybe server to server integrations or it can be manual imports as well. Now, when this happens, when we collect all this data, we have JavaScript tags and SDKs.
Speaker 1: Or maybe server to server integrations or it can be manual imports as well. Now, when this happens, when we collect all this data from different various sources, we are going to remap it because we want that data in a particular format. Speaker 1: And then in the end, we are going to store this data into the target database. Now, this can be further useful to create a single customer view, which is basically what CDPs do.
Speaker 1: They aggregate your user profiles into a single customer view, which means you can view all data of one individual customer at one place. Speaker 1: So this is often this is often basically a holistic representation of your customer data. Speaker 1: And then businesses can leverage this to learn how to better interact with customers or tailor marketing messages.
Speaker 1: All the recommendation system that you see on Instagram or other social media app, they are basically a part of this basically just evolves from this. Speaker 1: Then moving next, what is observability? My favorite topic, because I've been working on this since two years. Speaker 1: And I feel it's super, super important to understand this and adopt this in all our softwares.
Speaker 1: So observability simply answers that why X is broken. Speaker 1: And when we get the answers to the questions, things become simple. That's so easy. Speaker 1: If something is broken and we know that why is it broken or something, you know, your website is not working. Speaker 1: The latency is very bad or maybe it's your website is kind of slow and you know that why is it happening?
Speaker 1: You actually know when or like you actually know how to fix it and you will immediately fix it. Speaker 1: That's what observability do. It's very simple now. Speaker 1: Like if you have simple systems that have fewer moving, moving parts, which make them easier to manage. Speaker 1: If you want to monitor CPU memory database and the other networking conditions, it is usually enough to understand the systems and apply the appropriate fix to a problem.
Speaker 1: But I will say that observability is better suited for the unpredictability of distributed systems. Speaker 1: When we have complex systems, we have a lot of microservices because it allows you to ask the question about your systems behavior as the issue arise. Speaker 1: And you can simply ask that why X is broken or why is causing latency right now?
Speaker 1: So these are the few questions which observability answers. Speaker 1: So observability is something we can say it's a practice which we have to adopt. Speaker 1: We will have to modify our softwares. We will have to adopt certain practices to get observability of your system. Speaker 1: If we talk about data ingestion, like one of the benefit of observability is that you are basically handling the data of customer and or the clients.
Speaker 1: So clients needs to know as well that what is happening when the data is moving forward through the data pipelines or why is it even failing? Speaker 1: So there's kind of a transparency, which we can ensure through adopting observability and data ingestion, which we'll talk in detail more about it in the later slides. Speaker 1: But yes, observability provides transparency. It tells you about all the failures and so that you can quickly, you know, mend it.
Speaker 1: So there are different benefits of observability, which I've talked about in this slide. Speaker 1: It's the better visibility. It's easy that you are able to, you know, it's quite visible that where your data is when it is moving through the data pipeline. Speaker 1: What's happening? Is it feeling or is it passing? What's the success rate? What's the failure rate? There's a better visibility.
Speaker 1: Better alerting. If something is breaking, you get alerted or if something is loaded, is getting loaded. Speaker 1: For example, there's a lot of traffic or a lot of data is suddenly uploaded into your data pipeline and your data pipeline is not capable. Speaker 1: Your infrastructure is not capable enough to hold that much data. Then you will quickly get alerted because now you know that memory is filling up.
Speaker 1: So better alerting so that you can quickly fix it. Better workflow. Observability allows developers to see request end to end journey. Speaker 1: They can see the end to end journey along with relevant contextualized data about a particular issue, which in turn streamlines the whole investigation and debugging process. Speaker 1: Less time in meetings. If we have a process, then definitely the time gets reduced. It's quite easy. It applies in a normal life as well. It applies everywhere. So observability, it provides your process.
Speaker 1: Then there's an accelerated developer velocity. If you don't have to waste time in investigating so much and if you have just a single area where you can see what's happening with your whole software journey, then it becomes easy, right? Speaker 1: Your developer, you can focus on building features. You can focus on if something is broken, you can quickly fix it. So it's basically accelerating your developer velocity.
Speaker 1: Then observability, according to Twitter's blog, and I really love their blogs. According to that, there are four pillars of observability, which is metrics, alerting traces and logs, and it provides deep visibility into distributed system. Speaker 1: And it allows teams to get into get the root cause of them, you know, multitude of issues. And eventually it improves your system performance.
Speaker 1: This is something which is a statement that observability of data pipeline. It helps you determine efficiency and consistency of any data which is uploaded. Speaker 1: Now, this is quite important. Why is observability important in data ingestion? We can figure out till now that why is it important, but let's actually discuss this in detail that why is it important. There's a large volume of data which you have, which will run through Speaker 1: your data pipeline and and it's obvious that it cannot be 100% error free. If we have large volume of data, it's just not possible. So if you're.
Speaker 1: So we need to ensure if you have incorrect data that should not affect your correct data. Speaker 1: And if there is any failure, your data pipeline is able to handle it gracefully. Your errors should not affect the whole process. So we should log so that we can take quick actions. Speaker 1: All this comes all this is a part of observability and this shows that why it is so important to have this process.
Speaker 1: Then we have this is maybe we can understand this in a better better exam. Like if we if we are talking about data, we are talking about business and if our data ingestion is delayed due to some X, Y, Z problem, maybe your system is not working. Your infrastructure is broken. Speaker 1: Or it could be anything. Then it's going to cause the problem in business as well, because your business analytic operation will be waiting for your result for the data ingestion output. And hence we can see that it's it's quite important to focus on this.
Speaker 1: Yeah, I've already talked about how it can affect your business eventually affecting your revenue. Also very important. If your data is incorrectly processed. Speaker 1: If your data is incorrectly processed, you can take wrong decision. Let's understand this in this way. For example, Speaker 1: Let's say you're working for a payment. You're doing the data ingestion for payment system. Let's say your company's payment system, and you have some data of different cities or different states. Let's say your data is not correctly processed for one city. And there is some decision making which evolves, which involves the data.
Speaker 1: For all the cities. And one of them is corrupted. Then definitely going to make a wrong analytics if your data is incorrect for the whole city. And this can eventually lead to a big loss. So we need to ensure that there's a proper process. Speaker 1: To track all these failures and this shows that how much important it is for the business revenue and, you know, important right decision making to have a proper process for observability.
Speaker 1: Forbes as well, these are some newspaper clippings. I just wanted to put this. I just wanted to show this that even in Forbes, it has been featured that data observability is quite important. And Forbes defined this as a set of tools that allows data and analytics team to track the health of enterprise data system, identifying troubleshooting and fixing the problems when things go wrong. In other words, data observability refers to an organization's Speaker 1: Ability to maintain the continuous pulse of their data systems of their data systems by tracking monitoring and troubleshooting incidents to minimize and eventually prevent the data issues, downtime, and basically overall improving your data quality.
Speaker 1: Yes. Cool. So how can we achieve it? Now let's talk about how actually we can achieve it. Now before achieving it, we might need to ask a few questions before building any solution for it. We need to ask a few questions about freshness. Is your data arriving on time? Is it up to date? We need to ask a question about volume. Did all data arrive or did it get corrupted or all the data tables Speaker 2: Are complete? The distribution? Where was the data delivered to? Lineage? What are the downstream ingestors and upstream sources of the given data and schema? Is this data in the correct format? Because if it's in the wrong format, it might cause failure in your whole data pipeline.
Speaker 2: So then we have, yeah, then we have, in the end, we have to build a custom solution. And what I see as a solution is, and what I've worked in my previous organizations as well, is having one dashboard where you can see all those errors and where you can see all those failures, where you can see all the error messages in your, whatever is happening when your data is moving to, like when your data is doing a journey in data pipeline.
Speaker 2: If you have one dashboard where you can track all this, then it's good. It saves you time. And it also creates transparency in your data pipeline. And you can also enable the alerting and can quickly take the action. Speaker 2: Well, some of the metrics which I feel are quite important and I've worked on is the volume of data processed for single runoff pipeline, total number of records processed, you have a lot of data. And if you can somehow, you can calculate the total number of records at the source of truth.
Speaker 2: And then you can calculate the total number of records when the whole data ingestion process happens, then you can create the efficiency, then you can calculate the efficiency of the whole process. And if there are so many records, which has been dropped. Speaker 2: Within the, you know, within the whole data data pipeline within the whole process of remapping and ingesting into the target database, if something is getting lost, then something is wrong. And then we need to take action.
Speaker 2: Now when you're so, so this will calculate your efficiency and you can track like when you're, you know, making the, doing the remedies are all these, you can track through the metrics that things are happening or not. Speaker 2: So these are some of the metrics which you can go through. We can have error rate, error messages. So this is what we can do. We can collect all the error messages and we can show it in our one dashboard.
Speaker 2: Cool. So, uh, we are almost, and we have just five minutes left and I just want to explain about one metrics, which I, uh, and I will, I would love, I would have loved to have done a demo, but we are short of time. Speaker 2: So I will, you know, paste all the links in the comment section. I, I will paste my GitHub link of the demo and, uh, I, and the best part is I've done, I've made the blog post as well.
Speaker 2: So if you have any questions, you can quickly, you know, you can just ask in that blog post in the comment section of that blog post. Speaker 2: So yes. So one of the metrics which I worked on was, uh, there was a use case in which there was a streaming source and I wanted to calculate total number of records for streaming sources.
Speaker 2: Uh, for this, I used, uh, this stack driver, uh, my, uh, whole data was there in Google cloud and, uh, stack driver is basically, um, stack drivers provided by Google, uh, by Google cloud. Speaker 2: And where, uh, I can, I can just, you know, understand from this, uh, from this diagram, we can see that a user wants to track all the number of, you know, all the, uh, pubs or messages, which are there for streaming sources.
Speaker 2: So it will quickly open the stack driver dashboard, which is there in Google cloud. Speaker 2: And then stack driver will call up some messages that, Hey, please send me the metrics. Speaker 2: And then you can see all your streaming sources, like all the, all the number of pubs or messages in real time, like all the data, which is getting ingested into the data pipeline in real time.
Speaker 2: You can eventually, uh, achieve this programmatically using the stack driver API. Speaker 2: And I, I've talked about this in detail in my blog. Speaker 2: I've also done a demo, which is here and yes, you can open there. Speaker 2: So yes, this is it. Speaker 2: Uh, this is it. Speaker 2: I would also love to share about the, um, I'll just quickly share the example with you.
Speaker 2: Yes. Speaker 2: I hope it is there. Speaker 2: So yes, this is the example. Speaker 2: You can check out this on GitHub stack driver example, and you can open this in get board. Uh, basically get board is an open source developer platform for remote development. Uh, so you will get everything prepared. Speaker 2: You will, you won't have to, uh, you know, just install anything.
Uh, you'll just get out of the box. Speaker 2: So you can quickly, uh, open this in get board and get your work done. Speaker 2: I have put all the instructions here. Speaker 2: It will be cool. You know, to see all your, uh, real time messages, you can, uh, track the metrics of your real time messages and it will give you the feel of how you can achieve, uh, different metrics for observability.
Speaker 2: So yes, uh, this is it about the talk. Uh, thank you so much, everyone. Speaker 2: I really enjoyed this talk. Uh, if you have any question, please personally ping me on Twitter or LinkedIn.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why every data ingestion pipeline needs observability
Nancy Chauhan argues that large data pipelines will always contain errors, so teams must design them to show what failed and why.
In short
Data pipelines that move large volumes of batch and streaming data will never be completely error free, so they need to be observable. Software engineer Nancy Chauhan explains what data ingestion is, why observability matters more than basic monitoring, and how a handful of questions and metrics, gathered in one dashboard, help teams catch failures before they lead to bad business decisions.
Key takeaways
Data ingestion
Data engineering
CDAP
Data monitoring
Why ingestion needs a control system
Chauhan opens with a simple premise: almost every organization now ingests, transforms and stores huge volumes of data, and business analytics depends on it. That makes the ingestion layer a place where teams need a control system that tells them whether the data is right. She frames observability as that control system.
"So observability, it will help you to determine the completeness, accuracy, efficiency and consistency for any data which is uploaded through either batch or streaming sources." (0:00)
She then walks through what ingestion actually involves. Data about a user arrives from websites, mobile apps and CRMs, collected through APIs, JavaScript tags and SDKs, server to server integrations or manual imports. It is remapped into a required format and stored in a target database. From there, it can feed a single customer view of the kind customer data platforms build, which businesses use to tailor marketing messages and power recommendations.
Observability answers why something is broken
Chauhan, who has worked on reliability for two years, gives the most compact definition in the talk.
"So observability simply answers that why X is broken." (4:47)
She distinguishes it from basic resource monitoring. For simple systems with few moving parts, watching CPU, memory, the database and network conditions is usually enough to diagnose a problem. Complex systems built from many microservices are different.
"But I will say that observability is better suited for the unpredictability of distributed systems." (5:37)
In her view, observability is a practice rather than a product: teams have to modify their software and adopt specific habits to get it. For data ingestion there is an added reason. Pipelines carry customer and client data, and those clients need to know what is happening to it and why it might be failing. Observability creates that transparency.
What teams gain
Chauhan lists a set of practical benefits. Better visibility shows where data is in the pipeline, whether it is passing or failing, and the success and failure rates. Better alerting warns a team when, for example, a sudden flood of data is filling memory faster than the infrastructure can cope. A better workflow comes from seeing each request end to end.
"Observability allows developers to see request end to end journey." (7:42)
She adds less time in meetings, because a clear process shortens investigations, and faster developer velocity, because engineers spend less time hunting for causes and more time building features. Citing Twitter's engineering blog, she names four pillars of observability: metrics, alerting, traces and logs.
Why it matters for the business
The heart of the argument is that large data volumes guarantee some errors. The goal is not perfection but containment.
"So we need to ensure if you have incorrect data that should not affect your correct data." (9:43)
Pipelines should handle failures gracefully and log them so teams can act quickly. Chauhan gives two business consequences of getting this wrong. Delayed ingestion leaves analytics teams waiting for results. Incorrect processing is worse: she describes a payments dataset covering many cities where one city's data is corrupted, so decisions that rely on all cities are skewed and can lead to a big loss.
"If your data is incorrectly processed, you can take wrong decision." (10:47)
She also points to Forbes coverage, as the speaker cited, which describes data observability as tools that let data and analytics teams track the health of enterprise data systems and troubleshoot problems when things go wrong.
Questions, metrics and one dashboard
Before building anything, Chauhan recommends asking five questions. Freshness: is data arriving on time and up to date? Volume: did all the data arrive, or was some corrupted? Distribution: where was it delivered? Lineage: what are the upstream sources and downstream consumers? Schema: is the data in the correct format, since a wrong format can break the whole pipeline?
Her answer, drawn from her previous roles, is a custom solution centered on a single dashboard showing failures and error messages across the data's journey, with alerting attached.
"If you have one dashboard where you can track all this, then it's good. It saves you time." (13:53)
The metrics she has found most useful include the volume of data processed per pipeline run, and the total number of records at the source of truth compared with the total after ingestion. That comparison gives the efficiency of the process: if records disappear during remapping and loading, something is wrong. Error rates and collected error messages complete the picture.
She closes with a streaming example. To count messages from streaming sources whose data lived in Google Cloud, she used Stackdriver, which shows message counts in real time on a dashboard.
"You can eventually, uh, achieve this programmatically using the stack driver API." (16:46)
Short on time for a live demo, she shares a Stackdriver example on GitHub that viewers can open in Gitpod without installing anything, plus a blog post covering the approach in detail.
Watch the talk
Observability In Data Ingestion, Nancy Chauhan, 18 minutes: https://www.womentech.net/video/observability-in-data-ingestion-nancy-chauhan
About the speaker
Nancy Chauhan is a Software Engineer from Bengaluru, India, who enjoys hacking through software engineering problems and has spent the past two years developing solutions for software reliability. In her free time she writes tech blogs on engineering topics, makes illustrations and goes on short cycling trips.
Questions this talk answers
What is data ingestion?
How is observability different from monitoring?
Why does data ingestion need observability?
What should a team ask before building a data observability solution?
Which metrics help observe an ingestion pipeline?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Four pillars of observability (metrics, alerting, traces, logs)
Data observability questions: freshness, volume, distribution, lineage, schema
Single customer view (customer data platforms)
Resources mentioned
Nancy Chauhan's blog (nancychauhan.in)
Stackdriver example on GitHub (speaker's demo repository)
Stackdriver API
Gitpod
Twitter's engineering blog on observability
Cite this talk
Nancy Chauhan. "Observability In Data Ingestion." WomenTech Network, 2022. https://www.womentech.net/video/observability-in-data-ingestion-nancy-chauhan