In short
Observability is what lets engineers answer the questions a distributed system raises when something goes wrong: which services a request passed through, where it failed and where it slowed down. Shweta Sharma, Senior Software Engineer at Intuit, argues that in microservice architectures this visibility has to be built in through three pillars (metrics, logs and traces) plus visualization, and she names open source tools that implement each one.
Key takeaways
- Understand what observability is and why we need it.
- Understand the three pillars of observability (logs, metrics and traces) in detail, and which tools and technologies you can use to implement them.
- Learn how to instrument your microservices to answer questions such as which services a request went through, why and where the request failed, and where the performance bottlenecks are.
Lessons from the Matrix: no system is perfect
Sharma opens with a film most engineers know. In the Matrix, Neo could read the system's patterns and predict the agents' moves, while the Architect could see the anomaly but could not eliminate it. She defines an anomaly as whatever breaks the pattern a system normally produces in its metrics, whether the trigger is the day of the week, the time of day or a program that happens to be running.
Two lessons follow. However hard a team tries to build a perfect application, failures will happen. And, as she puts it:
"it is hard to fix even a known bug or issue in a complex and old system" (3:01)
From monolith to microservices
Sharma contrasts the monolith, where the UI, business logic, back end and database live in one indivisible unit, with microservices, where each service has a single responsibility: products, inventory, sales, accounts. The split lets a system grow with the business, but it changes how failures behave. With services running on Kubernetes, shipped through CI/CD pipelines and developed at high speed, the number of possible failure modes multiplies.
"when something fails it is no longer obvious what causes just like architect could not figure out in this complex system where this anomaly neo came from and why it was happening" (5:04)
Observability, in her framing, is what answers the follow-up questions: which API is adding latency, which one is failing, how each microservice handled the request and when it failed. To get there, a system needs logs, metrics and traces, plus visualization so the data is readable without extra effort.
Metrics: finding where the problem is
Sharma describes metrics as values about a system, usually counts or gauges aggregated over time as time series. She separates two families. USE metrics (utilization, saturation, errors) describe the system: memory, CPU, database connections, the health of each pod. RED metrics (rate, errors, duration) describe the application: requests per second, errors, failed customer interactions (FCIs) and latency.
The value comes from connecting the two. When a customer experience breaks:
"application metrics will tell you which api or service contributes to that" (9:57)
System metrics may then show dropping database connections, which points to the database itself being down, even when another team manages the infrastructure. For Java and Spring Boot services, she walks through Spring Boot Actuator exposing metrics and Micrometer acting as a facade that pushes or pulls them to a monitoring system. She recommends Prometheus because it is open source and pulls data periodically over HTTP, with Grafana or Wavefront for visualization. Spring Boot also provides many metrics out of the box, including HTTP requests, JVM memory, garbage collection, thread counts and Kafka and Hikari metrics.
Logs and traces: what went wrong and where the time went
Metrics locate the problem. Then:
"now we have to find out what the problem is so you need more details right and that's where the logs come in" (12:32)
Sharma insists that logs be structured. A line that only says an error occurred on a date is of little use. A structured log records when the error happened, what it was, where it happened and whom it affected, so an engineer can later search for a user or API across a date range and see how many users or applications were impacted.
Logs have a limit in distributed systems, though. Latency can rise while every service's logs look healthy. Her answer:
"how about we add a request id or unique id in all the different applications or micro services which will link our call made to all the different services" (15:22)
That shared ID produces a trace: the full path of one request, broken into spans for each service call or unit of work. Spans show where most of the time is spent and where more instrumentation is needed. Traces also give context to the other pillars, pointing to which metrics and logs matter for a given issue. For tooling she highlights OpenTelemetry and Jaeger, with global attributes such as environment name and request ID and custom ones such as product or user ID. She demonstrates Lightstep, a paid visualization tool and OpenTelemetry contributor, showing a failing profile request, the services involved and the database queries adding latency.
Observability versus monitoring
Sharma closes by untangling two terms that are often confused:
"monitoring is what you do after a system is observable" (20:01)
Monitoring covers the known: the metrics, alerts and dashboards a team already knows it needs. Observability addresses the unknowns, such as noticing that a call rate which is usually steady has jumped tenfold, a sign that something is about to go wrong even though nothing has failed yet.
Watch the talk
"Observability in today's complex systems" by Shweta Sharma (23 minutes): https://www.womentech.net/video/observability-in-todays-complex-systems-shweta-sharma
About the speaker
Shweta Sharma is a Senior Software Engineer at Intuit who started her career at a startup building a digital marketing product. She later focused on cloud technologies and architectures at Intel and worked with Microsoft and Pandora before joining Intuit.
Questions this talk answers
- What is observability?
- What are the three pillars of observability?
- What is the difference between USE and RED metrics?
- How do traces help when logs show nothing wrong but latency rises?
- How is observability different from monitoring?
Drafted with AI from the talk transcript. Editor review pending.