What is observability? Seeing inside a running system

Sandeep Patil’s wall at the Baner office is covered in dashboards, screens of live charts, and every one is green. They were green before the last outage, too. To measure the SLO, he needs to see inside SeatJaldi.

On Sunday, during a match-day sale, bookings quietly fail for a stretch. CPU, how busy each server’s processor is, sits at 40%. No alert goes off, so nothing calls a person. Meera Kulkarni hears it from users on a social app.

Meera asks why nothing warned them, and why bookings failed. Arjun Nair adds six dashboards and an alert for CPU over 80%. None would have caught Sunday.

“Green walls,” Sandeep says. “Very calming. Also very wrong.” He pulls up one complaint: a fan tapped Book and saw an error. So where did that one booking go wrong?

Observability is being able to ask a running system new questions, and answer them from what it already sends out. If it sends out enough, a new question needs no new code.

A doctor works the same way. A patient says “I feel unwell”, and the doctor asks unplanned questions using three kinds of data. The first is a number taken again and again, like an hourly temperature.

The second is a written record of one thing that happened, like a line in the patient’s diary.

The third follows one request from service to service, like dye moving through the body on a scan.

Keep one line in your head: numbers tell you something is wrong, logs tell you what, traces tell you where.

• Metric: is something wrong? Like the fever on the chart.
• Log: what happened? Like the line in the diary.
• Trace: where did it go wrong? Like the organ on the scan.

What just happened? Trends are metrics, one event’s words are a log, and where the time went is a trace.

• A food app shows “Error code 7F3A”, and support asks for it. That code finds your request’s logs and trace.
• A status page says “All systems operational” while your tap fails. It tracks servers, not users.
• A support agent sees your order stuck at payment. A trace or log showed where.

A metric is a number over time, a log is a record of one event, and a trace follows one request across services. You have met all three, so now look inside a trace. It times each stop separately, like each organ on the scan.

Here is Sunday’s failed booking, BK-10473, as a trace. A trace id, a short code passed from service to service, ties its spans together.

What just happened? The spans add up to 30 + 2,000 + 10 = 2,040 ms, plus the booking service’s own 10 makes 2,050. Payment took about 98%. On the scan, one organ lights up red.

The four golden signals are latency, traffic, errors and saturation. Watch them for every service users touch. If you can measure only four things, measure these.

The answer to the pause: watch errors and latency, the symptoms users feel. A busy CPU is a cause, one possible reason why. Think of the nurse again. She calls the doctor for a patient in trouble, not for a monitor beeping in an empty room.

What just happened? On Sunday the booking service sat waiting for payment. Waiting uses almost no CPU, so CPU stayed at 40%, while errors would have jumped within minutes.

Engineers put the same ids on every signal, alert on symptoms and budget burn, and keep metric labels small. They add the code that sends these signals early, before any outage. Adding that code is called instrumenting.

First, a patient id on every diary line. Each log line carries the booking id and the trace id, so a log and its trace can be matched. A structured log writes each detail as a named field, such as booking_id, so tools can search by it.

What just happened? The log says what failed. The trace id opens the matching trace, and the booking id finds the complaint. In a real log file this sits on one line; it is spread out here to read.

Second, page on budget burn. To page means to ring the on-duty engineer’s phone, at any hour. The SLO allows 0.1% bad bookings, failed or over 1 second, over 28 days.

Page when two windows agree (assumed): bad bookings average 1%, ten times the allowed rate, over the hour and last 5 minutes. A 5-minute window alone pages on blips that barely touch the budget.

28 days is 28 × 24 = 672 hours. At the allowed 0.1%, a full budget lasts all 672. Ten times faster, it empties in 672 / 10 = 67.2 hours: urgent. At 0.2%, twice the rate, it lasts 336 hours, or 14 days, so a task for working hours will do.

Third, keep labels small. A label is a tag on a metric, such as the route (which endpoint was called) or the status code. Each mix of label values is its own time series, one line on the chart.

3 routes × 4 status codes = up to 12 series. Add a user_id label for 1 crore users: up to 12 × 1 crore = 12 crore series. Keep user ids in logs and traces, never in metric labels.

OpenTelemetry is an open standard that many teams use to collect and send these signals. It also covers baggage: small values a request carries between services. Treat metrics, logs and traces as the minimum, not the whole list.

• Track failed requests’ latency separately. An error that comes back at once makes latency look better than it is.
• A probe that tries the booking path as a user would catches what internal numbers miss.
• To test the wiring, fail one test booking on purpose, then find its log line and trace by booking id.
• CPU still has a place: on a service limited by its processor, it is one saturation measure on the dashboard.

Try it yourself

1. 1

   Name the four signals

   For the booking API, write one measure each for latency, traffic, errors and saturation.

2. 2

   Add a threshold

   Give each a number that makes a person act. Then pick the ones worth a page.

3. 3

   Mark the cause

   Keep CPU off your pages. Explain why in one line.

4. 4

   Check your answer

   You are right if you page on errors and latency, such as errors over 1% for an hour or p99 over 1 second. Paging on saturation near its limit is also fair. CPU stays off because it can be low while bookings fail.

Back to Sunday. Sandeep opens the trace for BK-10473. The payment span took 2,000 of 2,050 ms and timed out, so the booking service gave up.

A metric, the booking error rate, said something was wrong. The log line for BK-10473, a payment timeout, said what. The trace said where. Sandeep now reads SeatJaldi like the chart at the foot of a patient’s bed.

The pager now rings when the booking error rate burns the SLO budget fast, not when CPU climbs. Then Sandeep frowns. The traces show something else: a support account refunded bookings it should never touch. Who is allowed to do what?