Day 5: Break the Agent

Everything fails, sometimes twice

So far every call worked. In production, the model API times out during a traffic spike, the order database drops a connection during a failover, and — worst of all — a tool succeeds but its response never makes it back. The agent then doesn't know whether the refund happened. This lesson breaks the agent on purpose and makes it survive: failure injection, retries with exponential backoff and jitter, a time budget, and a graceful fallback, all without ever refunding twice.

What you build

supportdesk/resilience.py:

• FaultInjector(rate) — with probability FAULT_RATE a call fails. For the model, a timeout or connection error before the call. For a tool, either before (timeout) or after — the tool ran, then the response was "lost". The seed is fixed (FAULT_SEED=7) so your numbers match ours.

• retry(fn, attempts, base, cap, budget_s) — retries only transient errors (TimeoutError, ConnectionError), waiting random(0, min(cap, base × 2ⁿ)) between tries ("full jitter"), and gives up early if the next wait would cross the time budget. Validation and business errors are never retried: a bad argument won't get better by asking again.

• supportdesk/chaos.py — plays the same 60 conversations twice under faults, once with retries off and once with 5 attempts, and reports success, latency, retries and duplicate refunds.

In the agent, both the model call and each tool call go through retry. If the budget runs out, the agent doesn't crash; it replies:

I'm having trouble reaching our order system right now. Please try again in a minute; nothing will be refunded twice.

The lost-response problem

Here's the failure that makes retries dangerous: