Day 3: Dashboards and one alert worth waking for

Debug Microservices with OpenTelemetry and Prometheus ~5 min read 40 min hands-on

Build a RED dashboard in Grafana and compare two alerts on the same traffic: one pages on noise, the other only on a real incident.

An alert that fires for every hiccup trains people to ignore it. An alert that never fires lets a real outage run for an hour. Today you add Prometheus and Grafana to ShopStream, build a RED dashboard for all three services, and compare two alerts on the same traffic: one that pages for any error, and one that pages only when customers are actually hurting.

What you build

  • Prometheus (port 9090) scraping /metrics on checkout, inventory and payments every 5 seconds, with alert rules in prometheus/alerts.yml.

  • Grafana (port 3000), provisioned from files: a Prometheus data source and a dashboard called ShopStream β€” RED with four panels β€” rate, errors (% 5xx), p99 and p50 duration, each split by service. Anonymous viewers can look; admin login is admin / shopstream.

  • Two alerts on checkout:

    yaml

RED in PromQL

RED β€” Rate, Errors, Duration β€” is the minimum dashboard for any request-driven service:

histogram_quantile estimates the percentile from bucket counts, so its precision depends on the buckets you chose in telemetry.py (5 ms up to 10 s). Rates are always computed over a window β€” here 1 minute β€” which smooths noise but also delays how fast a change shows up.

The experiment

Traffic runs at 20 checkouts per second throughout:

  1. Noise (60 s): payments fails 1% of requests β€” the normal background of retries, deploys and flaky networks.

  2. Quiet (45 s): no failures, so every alert can resolve.

  3. Incident (60 s): payments fails 30% of requests.

The demo records every time each alert starts firing (a page) and in which phase.

What we measured

3,704 checkouts, 345 failed:

AnyCheckoutError woke someone up for 1% background noise and stayed red for 35 seconds of it. CheckoutErrorRateHigh ignored the noise and paged once for the real incident.

Read the full lesson free

This lesson is one of the first 3 in its course, free with an account β€” the full article, implementation guide and scripts. No card needed.

Already a member? Sign in