Day 4: Never lose an order

A customer clicks "Place order", order-service says "done", and the order never reaches the warehouse. Nothing crashed in your code; a broker did, and the producer had been told the write was safe when it was not. Today you run three brokers, kill the leader in the middle of 20,000 orders, and count exactly how many acknowledged orders vanish or arrive twice under three producer settings.

What you build

• A 3-broker KRaft cluster on ports 19092, 19192 and 19292. Every partition of orders has 3 replicas and min.insync.replicas=2.

• Three producer configurations in ProducerModes, so the same burst can be replayed with each:

• POST /bursts publishes a paced burst (here 20,000 orders at 5,000/s) and records which order ids the brokers acknowledged. Those ids are promises.

• fulfillment-service counts, per run, events received and distinct order ids. Lost = acknowledged but never delivered. Duplicates = delivered more than once.

The drill

After the burst, the broker is restarted, the demo waits until every partition is fully replicated again, lets the consumer settle, and compares the acknowledged ids with the delivered ids.

What we measured

In the browser sandbox, one kill per mode:

The killed broker was back with all replicas in sync about 20 s after the kill each time (19.1–21.6 s).

Read the first row carefully: the producer got no errors. All 20,000 sends succeeded. Yet 65 orders that the brokers acknowledged never reached fulfillment.