Day 7: Run the Evaluation Harness

"It seems to work" is not a test

Every lesson so far ended with a demo. Demos are chosen by the person who wrote the code. Before an agent talks to customers, you want a fixed, labelled set of conversations, a score you can compare between versions, and a hard failure on the one thing support agents must never do: state something false about a customer's order. This lesson builds that evaluation harness and runs it against everything you built.

What you build

supportdesk/evals/:

• cases.json — 50 labelled conversations covering the whole system: 10 order-status phrasings, 5 two-turn "where is my order?" chats, 9 partial refunds (3 over two turns), 4 large refunds that must be held, 5 orders that can't be refunded (placed or cancelled), 7 stock checks, 3 policy questions, 2 questions that need an order id, and 5 off-topic requests. Each case says what "done" means (check) and which tools should be called, in order (expect_tools).

• run_eval.py — replays every case on a fresh database and scores:

• Task success — the reply does what the check requires (e.g. contains the order's real status, or a refund number, or "review" for a held refund).

• Tool selection accuracy — the exact tool sequence matches.

• Hallucination rate — the reply mentions an order id the customer never gave, states a status that isn't the real one, or quotes a stock number that doesn't match the database. Truth comes from SQLite at eval time, not from the case file.

• Latency p50/p99 and estimated cost per 1,000 conversations (from Day 6's telemetry).

• It writes report.json and a readable report.md, and exits non-zero unless task success ≥ 90% and hallucinations are 0%.

The first run failed — usefully

The first full run scored 94% with three failures: