Day 6: Build an Observability Wrapper

You can't fix what you can't see

The agent now works, defends itself and survives failures. In production the questions change: how slow is it at p99, how often do tools fail, how many tokens does a conversation use, and what will it cost at 10,000 conversations a day? Logs won't answer those. This lesson adds telemetry: one trace per request with every model and tool step, stored and aggregated into a /metrics endpoint.

What you build

supportdesk/telemetry.py:

• start(session_id) creates a trace with a trace_id (in a thread-local, because the server handles requests on many threads).

• step(kind, name, started, ok, ...) records one model or tool call: latency, success, and for model calls the input/output tokens.

• finish(...) writes one row to a new interactions table: latency, model calls, tool calls, tool errors, retries, tokens, estimated cost, and the steps as JSON.

• metrics() aggregates all interactions: success rate, p50/p99 latency, tool success rate, retries, tokens and cost per 1,000 conversations.

The agent calls step() around every model decision and tool call — including failed ones — and finish() at the end of run(). The API gains:

• GET /metrics — the aggregates above.

• GET /traces/TRACE_ID — every step of one request.

• An X-Trace-Id header on every /agent response, so a support ticket can quote the exact trace.

Tokens and cost without a paid model

The offline model doesn't bill anything, but you still want to know what a hosted model would cost. The offline model estimates tokens as characters ÷ 4 (close enough for English JSON), and the cost uses PRICE_IN_PER_MTOK and PRICE_OUT_PER_MTOK (default $3 and $15 per million). With ANTHROPIC_API_KEY set, the real usage numbers from the API are used instead. Either way the formula is the same: