Day 6: Run it like production

The code works. Now it has to survive a Tuesday: a malformed message from a partner integration, a consumer that falls behind during a deploy, a broker that dies at 2 a.m. Today you add the operational pieces every Kafka system needs — retention, a dead-letter topic, a lag alert — and run a broker failure drill against the 3-broker cluster from Day 4 to measure how long recovery really takes.

What you build

• Retention on the orders topic: 7 days (retention.ms=604800000), set in Topics.java and checked through the AdminClient at GET /ops/topic.

• A dead-letter topic, orders.DLT (30-day retention). Spring Kafka's DefaultErrorHandler retries a failing record twice, 200 ms apart, then publishes it to orders.DLT with the exception in its headers and moves on. A record that cannot even be parsed is not retried at all.

• A lag alert with a "for" duration, like a Prometheus rule: OK → PENDING when lag passes 500 → FIRING if it stays above 500 for 5 s → OK when it drops back.

• A failure drill: 200 orders/s for 20 s with the safe producer, kill -9 the leader of partition 0 after 5 s, then measure the producer stall, the consumer lag and the time until all replicas are back in sync.

The dead-letter topic

Without this, a single unreadable record ("poison pill") is retried forever and its partition stops moving — every order behind it waits. With it, the bad record is parked with its original partition, offset and exception class, and the consumer keeps going. Note the explicit destination: Spring Kafka 3.3 would otherwise publish to orders-dlt, a topic this cluster does not auto-create. We found that the hard way in the first run.

The alert

The "for" duration is what makes an alert worth waking up for: a deploy that causes a 3-second spike never fires. The evaluator runs in order-service because that service stays up while fulfillment is down — in production this lives in Prometheus or your monitoring system, fed by the same lag numbers.

What we measured

In the browser sandbox: