Day 6: Move slow work off the request path

After an order is placed, ShopStream emails a receipt. The email provider takes 300 ms and sometimes fails. Sent inside the request, that's 300 ms added to every checkout, and a failed email either fails the order or is silently lost. In this lesson you move receipts onto a queue — a Redis Stream — so checkout returns in under a millisecond. Then you try hard to break the workers: random provider failures, a message that can never succeed, and a worker that crashes at the worst possible moment. Every receipt still goes out exactly once.

The problem

The rule of thumb: do in the request only what the response depends on. The shopper needs to know the order is placed. They don't need to wait for an email to be accepted by a third party. Work like that — emails, thumbnails, search indexing, webhooks, analytics — belongs on a queue, processed by workers at their own pace.

A queue raises its own questions, and they're the ones interviewers ask: What if a worker crashes mid-job? What if a job fails forever? Can a customer get two emails?

What you build

• receipts.py — send_receipt(order) simulates the provider: sleeps SEND_MS (300 ms) and fails with probability FAIL_RATE. Orders from user poison always fail. enqueue(order) adds a job with XADD receipts * ….

• app.py — after the order is placed: with RECEIPTS=inline, send the email in the request; with RECEIPTS=queue (the default), XADD and return. GET /queue shows waiting jobs, pending jobs and dead letters.

• worker.py — a consumer in the group mailers.

• checkout.py — times 40 checkouts. queue_demo.py — the failure test.

Why Redis Streams

A Redis list with LPUSH/BRPOP is a queue, but once a worker pops a job, Redis forgets it. If the worker crashes, the job is gone. A stream is an append-only log with consumer groups: each job is delivered to one consumer in the group, and stays in the group's pending entries list (PEL) until that consumer acknowledges it with XACK. Crashed? The job is still pending and another worker can claim it. Kafka works the same way with offsets, at much larger scale — that's the next course.

The worker loop

Three ideas are in that loop: