Skip to content
POLARISAI ACADEMY
R03-401 · Navigator

Observability, Incident Response and Postmortems

You can write one JSON log line, follow one correlation id across three services, read p50, p95, and p99 from a sorted tape of 20 latencies, and compute the error budget, MTTD, and MTTR for a checkout outage.

Level
Navigator
Time
25 h (estimate)
Prerequisites
R03-301
Course 4 of 4
Jump to

After this course you can

The student can instrument a service with correlation IDs, structured logs, and SLO error-budget alerts, trace a seeded production fault to its exact root cause, and restore service via runbook automation.

One JSON object and four levels

A structured log stores one event as one JSON object, with the same field names on every line. The fields in this lesson are ts, level, service, correlation_id, msg, status, and latency_ms. When a timeout ends the call, the line also has timeout_ms. ts is the finish time, written as hours, minutes, seconds, and milliseconds. status is the HTTP status this service returned to its caller. latency_ms is the whole milliseconds from the start of this service's work to ts. A pay line from the outage reads {"ts":"09:12:03.300","level":"ERROR","service":"pay","correlation_id":"c7f3","msg":"upstream timeout","status":504,"latency_ms":200,"timeout_ms":200}. Count the names: that line has 8 fields. The same call without timeout_ms would have 7. A prose sentence such as pay failed gives you the words and no field you can filter on.

The level is one of four words, and you pick it in this order. If status is 500 or higher, or the call stopped because timeout_ms was reached, the level is ERROR. A status of 504 with latency_ms 200 is ERROR even though 200 is under the 500 millisecond bar, because the status rule comes first. If status is 200, 201, or 204 and latency_ms is over 500, the level is WARN. If status is 200, 201, or 204 and latency_ms is at most 500, the level is INFO. DEBUG is a line you add on purpose, such as the full request body, and you leave DEBUG off while the service is taking traffic. Apply that order to two tape values from the next section. Status 200 with latency_ms 180 is INFO. Status 200 with latency_ms 900 is WARN.

The same correlation id on every hop

The gateway mints one correlation id for an incoming checkout and sends that id to the next service in a header. Checkout and pay copy the same id onto their own lines. A new id on the next hop would split one call into two searches. For the call in the example the id is c7f3. A search for c7f3 returns three lines: gateway, checkout, and pay. The pay line is the one that carries timeout_ms, and that number is what the process compared with the clock.

In these lines the correlation id is also the trace id. A trace is the set of spans that share that id. The span fields used here are the ones OpenTelemetry records for a hop: a span id, a parent, a start, and an end. Duration is end minus start, in milliseconds. The gateway span id is g1 and it has no parent, so g1 is the root. The checkout span id is c1 and its parent is g1. The pay span id is p1 and its parent is c1. Gateway runs from 09:12:03.000 to 09:12:03.400, so its duration is 400 ms. Checkout runs from 09:12:03.040 to 09:12:03.360, so its duration is 320 ms. Pay runs from 09:12:03.100 to 09:12:03.300, so its duration is 200 ms, the same figure as timeout_ms. The caller wait is the root duration, 400 ms. Adding 400, 320, and 200 counts the nested pay work again and produces 920, which is longer than the caller waited. Own time is the span duration minus the child duration. Checkout own time is 320 minus 200, which is 120 ms. Gateway own time is 400 minus 320, which is 80 ms. Pay has no child, so its own time is 200 ms. Those three own times add to 80 plus 120 plus 200, which is 400 ms.

Percentiles, throughput, and RED

Sort the latencies from small to large and give the first one rank 1. With the nearest-rank rule, the p-th percentile is the value whose rank equals the ceiling of p/100 times N, where N is how many latencies you sorted. The tape from the day before the outage has N = 20 and is already sorted, in milliseconds: 80, 90, 100, 110, 120, 130, 140, 150, 160, 180, 200, 220, 240, 260, 300, 340, 400, 480, 900, 2400. For p50, 0.50 times 20 is 10, and the ceiling of 10 is 10, so p50 is the 10th value, 180. For p95, 0.95 times 20 is 19 exactly, so p95 is the 19th value, 900. For p99, 0.99 times 20 is 19.8, and the ceiling of 19.8 is 20, so p99 is the 20th value, 2400. Eleven values are at or under 200, from 80 through 200, and 11/20 is 0.55. Nine values sit above 200, and 9/20 is 0.45. Eighteen values are at or under 500, from 80 through 480, and 18/20 is 0.90.

Throughput is finished requests divided by the length of the window. From 09:10 to 09:46 the checkout counter shows 1,800 finished requests. That stretch is 36 minutes, and 1,800 divided by 36 is 50 requests each minute. 50 divided by 60 is 5/6 of a request each second, which rounds to 0.83 when you keep two decimals. RED is three numbers about the request path. Rate is the throughput, 50 per minute. Errors is the failed share: 810 of the 1,800 returned status 504, and 810/1,800 simplifies to 9/20, which is 0.45. Duration for the normal shape is the day-before tape, p50 180 ms and p95 900 ms. During the outage a call that hits the timeout finishes at 200 ms, so those 810 lines show latency_ms 200. USE is three numbers about one machine. In this example the pay process kept its one CPU busy for 400 ms of each second, so utilization is 0.40. The line of waiting requests peaked at 6, so saturation is 6. The process recorded 0 crashes, so the USE error count is 0, while the 810 results with status 504 stay in the RED error count.

Budget, probes, and the incident clock

A finished request is good when status is 200, 201, or 204 and latency_ms is at most 500. The SLI is good requests divided by finished requests in the window. The SLO is the target you choose for that SLI. This service uses an SLO of 99% good in each window of 10,000 finished requests. The error budget is the count of bad requests that target still allows: 1% of 10,000 is 100. An SLA is a customer agreement that names a consequence when an SLO is missed. The figures here are an SLO only, and no consequence amount is attached. After 810 bad requests the remaining budget is 100 minus 810, which is −710. The burn multiple is 810 divided by 100, which is 8.1, so the window has used 8.1 budgets.

A liveness probe calls /live and returns 200 when the process answers. A readiness probe calls /ready and, under the rule used here, returns 200 only when timeout_ms is at least 500, the same bar as the SLO. At the 09:15 check and at the 09:30 check, /live returned 200 and /ready returned 200. The pay lines in that same stretch show timeout_ms 200, so a /ready check that read timeout_ms would have returned 503. The check that actually ran only opened a socket. Synthetic monitoring sends a scripted checkout every 15 minutes, at 09:00, 09:15, 09:30, and 09:45. That page waits for two failed synthetics in a row. A faster rule pages when any 5-minute slice holds at least 25 bad requests. At 50 requests a minute and an error share of 0.45, one minute holds 50 times 0.45, which is 22.5 bad requests. Reaching 25 bad requests takes 25 divided by 22.5, which is 10/9 minutes, about 1.11.

Severity uses those same counts. SEV-1 means checkout is returning status 504 to callers and the bad count in the open window is already above the budget of 100. SEV-2 means a 5-minute slice has at least 25 bad requests and the remaining budget is still above 0. SEV-3 means a single ERROR line while the 5-minute bad count stays under 25. This outage is SEV-1, because 810 is above 100 and the caller-facing status is 504. Alert fatigue is what you get when a page route cries so often that someone mutes it. The night before, the fast route sent 30 pages. 28 of the 30 were one client retrying a status 404, and 2 of the 30 were real checkout failures. 28/30 rounds to 0.93. The route was then muted, so the next morning the fast rule stayed silent.

Three timestamps run the clocks. Impact start is the first caller-facing status 504, at 09:10. Detection is the page, at 09:30, when the second synthetic failed. The 09:00 synthetic had passed, the 09:15 synthetic failed without paging, and the 09:30 synthetic failed and paged. Restore is the first later checkout with status 200 and latency_ms at most 500. That line is at 09:46, correlation id c8b1, status 200, latency_ms 160, after commit a91c was rolled back. 160 is at most 500, so the level rule makes that line INFO, and restore stands at 09:46. MTTD is detection minus impact start, 20 minutes. MTTR is restore minus impact start, 36 minutes. Detection to restore is 16 minutes. A blameless postmortem writes the impact, the change, the detection gap, and the next action, and it names no person as the cause. Impact: 810 of 1,800 checkouts returned 504 from 09:10 to 09:46, and the remaining budget is −710. Change: commit a91c at 09:03 set timeout_ms from 3000 to 200. Detection: the fast rule was muted, and the synthetic page waited for a second failure, so MTTD is 20 minutes against a fast-rule figure of about 1.11 minutes. Action: /ready returns 503 when timeout_ms is under 500, and the fast rule stays unmuted at 25 bad requests in 5 minutes. The runbook you can follow on the next page has five steps. Copy the correlation id from the newest ERROR line. Pull every line with that id and read timeout_ms on the pay line. Compare it with the tape: p50 is 180, which still finishes under a timeout of 200, and 9 of the 20 values sit above 200. Roll back a91c so timeout_ms is 3000 again. Mark restore only on a new line with status 200 and latency_ms at most 500.

An example

From c7f3 to the two clocks

Walk one call, then the window around it. At 09:12:03.300 the pay line for c7f3 is the 8-field object above: level ERROR, status 504, latency_ms 200, timeout_ms 200. At 09:12:03.360 checkout logs the same id with level ERROR, status 504, and latency_ms 320. At 09:12:03.400 gateway logs the same id with level ERROR, status 504, and latency_ms 400. Each level is ERROR because each caller received status 504. The root span g1 is the caller wait, 400 ms. Own time is 80 ms in the gateway, 120 ms in checkout, and 200 ms in pay. 80 plus 120 plus 200 is 400. The sum of the three durations, 920, counts the inner spans again.

Commit a91c at 09:03 is the change that set timeout_ms from 3000 to 200. Impact start stays 09:10, the first caller-facing 504, which is a different id from the sample call c7f3 at 09:12. The 09:15 synthetic returns 504 and does not page. The 09:30 synthetic returns 504 and pages, so detection is 09:30 and MTTD is 20 minutes. Rollback finishes, and at 09:46 the line c8b1 shows status 200 and latency_ms 160. That is restore, and MTTR is 36 minutes. Across the 36 minutes the counter shows 1,800 finished requests, 50 each minute, and 810 with status 504. 810 divided by 1,800 is 0.45. The budget for a 99% SLO over 10,000 requests is 100. Remaining budget is 100 minus 810, which is −710, and 810 divided by 100 is 8.1. On the day-before tape, p50 is 180, p95 is 900, and p99 is 2400. Eleven of those 20 values are at or under 200, so the share that still finishes under the new timeout is 0.55, and the share that hits it is 0.45.

The postmortem keeps those figures and leaves out any person's name. Impact is 810 of 1,800 checkouts at status 504, remaining budget −710. The change is commit a91c and timeout_ms 200. The detection gap is a muted fast rule, 28 noise pages out of 30 the night before, plus a synthetic that pages on the second failure, so the page arrives 20 minutes after impact while 25 bad requests take about 1.11 minutes at 22.5 bad requests a minute. The action is a /ready result of 503 whenever timeout_ms is under 500, and an unmuted page at 25 bad requests in 5 minutes. The outage is SEV-1 because the bad count 810 is above the budget 100 and callers are receiving status 504.

Exercise

A second tape and a shorter outage

The new tape is already sorted. N is 20. The latencies in milliseconds are 100, 120, 140, 160, 180, 200, 220, 240, 260, 280, 300, 320, 360, 400, 440, 500, 600, 700, 1000, 1500. With the same nearest-rank rule, write p50, p95, and p99. Count how many values are at or under 500, and write that share with two decimals. The SLO is 99.5% good over 20,000 finished requests, and 40 of them are already bad. Write the budget and the remaining budget. Impact starts at 14:00, the page fires at 14:08, and restore is at 14:20. Write MTTD and MTTR in minutes. Throughput is 40 finished requests each minute and the error share is 0.20. The fast rule pages at 10 bad requests. Write how many minutes that takes. Then label two finished calls with the level rule: status 200 with latency_ms 1000, and status 504 with latency_ms 200.

Rank 10 is 280, so p50 is 280. Rank 19 is 1000, so p95 is 1000. Rank 20 is 1500, so p99 is 1500. Sixteen values are at or under 500, from 100 through 500, and 16/20 is 0.80. A 99.5% SLO leaves 0.5% bad. 0.5% of 20,000 is 100, so the budget is 100 and the remaining budget is 100 minus 40, which is 60. MTTD is 8 minutes. MTTR is 20 minutes. Bad requests each minute are 40 times 0.20, which is 8, and 10 divided by 8 is 1.25 minutes. Status 200 with latency_ms 1000 is WARN, because the body is usable and 1000 is over 500. Status 504 with latency_ms 200 is ERROR, because the status rule comes before the latency bar.

Check questions

Answer on your own first. Then open a question to see the answer.

On the day-before tape of 20 latencies, what are p50, p95, and p99 in milliseconds?

180, 900, and 2400. The ranks are 10, 19, and 20.

What is the error budget for a 99% SLO over 10,000 finished requests, and what remains after 810 bad requests?

The budget is 100. Remaining is 100 minus 810, which is −710.

For correlation id c7f3, which span is the caller wait, how many milliseconds is that wait, and why is 400 plus 320 plus 200 the wrong figure?

The gateway span g1 is the caller wait, and it is 400 ms. The sum 920 counts the checkout span and the pay span again inside their parents.

What are MTTD and MTTR, in minutes, when impact start is 09:10, detection is 09:30, and restore is 09:46?

MTTD is 20 minutes. MTTR is 36 minutes.

Take with you

One correlation id collects a call's JSON lines and spans, and the caller wait is the root span alone. Nearest-rank percentiles, the bad-request budget, and the two clocks are figures you can recompute from the sorted tape and the three timestamps.

Final task

Complete a keyed incident forensics investigation over a frozen telemetry dataset (identifying exact correlation ID, root-cause commit, MTTD, MTTR, and error-budget burn) and submit instrumentation code passing 25 observability tests. (Optional Portfolio item: written blameless postmortem).

The final task is marked against fixed answer keys and automated tests once the tests open.

Terms in this course

12 terms in the glossary belong to this course.

More lessons coming

The first lesson is open to read. The rest of the course is being written, and tests and diplomas come later. Join the waitlist