Day 6: Write the postmortem

Debug Microservices with OpenTelemetry and Prometheus ~5 min read 40 min hands-on

Run a full incident β€” bad deploy, page, rollback, recovery β€” and generate the postmortem with its timeline and numbers from telemetry.

Every incident ends with the same question: what happened, when, and what do we change? Teams that answer it from memory get vague timelines and blame. Teams that answer it from telemetry get exact times, real numbers and action items that stick. Today you run a full incident against ShopStream β€” a bad deploy, a page, an on-call response, a rollback, recovery β€” and generate the postmortem from the traces, metrics and logs you built in Days 1–5.

What you build

tools/incident.py, which plays every role in the incident and keeps the clock:

  1. Traffic: 20 checkouts per second, as in production.

  2. The bad deploy (T+0): inventory starts leaking 10% of its database connections (the Day 4 fault).

  3. Detection: the script polls Prometheus' alerts; the first page marks detected.

  4. Response: 20 seconds for a human to acknowledge and diagnose, then mitigation β€” roll back the change and recycle the pool.

  5. Recovery: checkout's error ratio over 30 s back under 1%.

  6. Evidence: peak error ratio and pool usage from Prometheus, one failing trace from Jaeger with its db_pool_acquire span, and the matching log line.

It then writes POSTMORTEM.md next to start.sh: summary, impact, a timeline with UTC times and T+ offsets, key numbers, evidence, root cause, what went well and badly, and action items. The format is blameless: it describes what the system and the process did, not who made a mistake.

The three numbers that matter

  • Time to detect (TTD): from the start of impact to the first page. Shortened by better alerts.

  • Time to mitigate (TTM): from the start to the moment the bleeding stops. Shortened by runbooks, rollbacks and feature flags.

  • Time to recover (TTR): until the system is healthy again by its own SLO. Includes everything above plus drain time.

Measure all three from the same clock as your telemetry, or the timeline will argue with your graphs.

Read the full lesson free

This lesson is one of the first 3 in its course, free with an account β€” the full article, implementation guide and scripts. No card needed.

Already a member? Sign in