Day 1: Build a Deterministic Dispatcher

15 min

Agenda

State Machine

Flowchart

Component Architecture

The Production Problem: Why direct non-deterministic LLM invocation wrecks p99 latency, inflates infrastructure bills, and introduces unbounded state failure.
System Architecture: High-speed edge pattern matching vs. slow-path LLM fallback.
Core Mechanisms: Regex slot extraction, parameterized SQLite execution, and response synthesis.
Trade-Off Analysis: Speed and determinism versus semantic flexibility.
Production Math: Latency, memory, and throughput profiling for zero-overhead routing.

The Production Problem: The Cost of Non-Determinism

In a production AI platform, calling a Large Language Model (LLM) is not a function call—it is an expensive, high-latency Remote Procedure Call (RPC) across the network to a non-deterministic black box. When e-commerce support platforms route every inbound user message ("Where is order ORD-1042?") directly to an LLM, two structural failures occur:

Tail Latency Amplification: Standard LLM inference introduces 800ms to 3,000ms of latency per call. For standard read-only lookups (e.g., checking an order status or checking inventory), this degrades p99 latency by up to 100x compared to a direct database query.
State Machine Corruption and Cost Runaway: LLMs can hallucinate schema formats, extract incorrect query parameters, or loop endlessly when attempting basic lookup routines. In high-volume systems, routing routine queries to non-deterministic models inflates token costs unnecessarily while multiplying failure rates.

Consider a real-world enterprise post-mortem: an e-commerce support provider experienced an incident where an unconstrained LLM agent entered an infinite tool-calling loop on ambiguous customer queries. The system generated over 14,000 redundant backend database calls per minute, exhausting connection pools and causing downstream cascading failures across the inventory service while generating thousands of dollars in wasted inference costs in under an hour.

To prevent this class of failure, high-throughput AI serving infrastructure relies on a Deterministic Fast-Path Dispatcher.

Code

 +-------------------------------------------------------+
 |                  Inbound HTTP Request                 |
 +-------------------------------------------------------+
                             |
                             v
 +-------------------------------------------------------+
 |              Deterministic Pattern Matcher            |
 |            (Regex Slot Extraction & Schema)           |
 +-------------------------------------------------------+
          |                                  |
     [Match Found]                      [No Match / Fallback]
          |                                  |
          v                                  v
 +-------------------+             +-------------------+
 | Parameterized SQL |             |  Slow-Path LLM    |
 |  Execution Layer  |             |  Inference Engine |
 +-------------------+             +-------------------+
          |                                  |
          +------------------+---------------+
                             |
                             v
 +-------------------------------------------------------+
 |            JSON Response (Intent, Reply, Latency)     |
 +-------------------------------------------------------+

Architecture of a Deterministic Dispatcher

The Deterministic Dispatcher acts as an edge router for customer messages. Before sending raw text to an external LLM, the dispatcher evaluates the payload against compiled regular expression rules and extracts structured tokens (slots such as order IDs or product names).

If a rule matches, the system routes the request immediately to an indexed SQLite query engine, constructs a deterministic output response, and returns in under 2 milliseconds. If no rule matches, the dispatcher flags the intent as unknown and forwards the request to the slow-path pipeline (which we will connect to LLM providers in later lessons).

Control Flow and State Transitions

Ingress: An HTTP POST /chat request delivers a JSON payload containing {"message": "..."}.
Intent Parsing & Slot Extraction:

Inspect input text against normalized regex patterns.
Extract target entities (e.g., pattern \b(ORD-\d{4})\b captures order IDs).

Data Access (Parameterized Query):

Map intent (order_status, refund_request, stock_check) to a pre-compiled SQLite query.
Bind extracted slots directly as query parameters to guarantee safety against SQL injection.

Synthesis & Egress:

Format query results into a structured human-readable reply.
Return {"intent": "...", "slots": {...}, "reply": "...", "latency_ms": 0.45}.

Key Implementation Snippets

Below are the core architectural mechanisms implemented in supportdesk.

1. The intent rules and slot extraction

The whole decision is a handful of compiled regular expressions in supportdesk/dispatcher.py. An order id wins first, because "refund" or "where is" only make sense together with an order:

python

ORDER_ID_PATTERN = re.compile(r"\b(ORD-\d{4})\b", re.IGNORECASE)
REFUND_KEYWORDS = re.compile(r"\b(refund|return|money back|cancel)\b", re.IGNORECASE)
STOCK_KEYWORDS = re.compile(r"\b(stock|inventory|available|in stock|have)\b", re.IGNORECASE)
POLICY_KEYWORDS = re.compile(r"\b(policy|rule|rules|returns|refunds)\b", re.IGNORECASE)

order_match = ORDER_ID_PATTERN.search(normalized_msg)
if order_match:
    order_id = order_match.group(1).upper()
    slots["order_id"] = order_id
    if REFUND_KEYWORDS.search(normalized_msg):
        intent = "refund_request"
        reply = self._handle_refund_request(order_id)
    else:
        intent = "order_status"
        reply = self._handle_order_status(order_id)
elif STOCK_KEYWORDS.search(normalized_msg):
    intent = "stock_check"

Anything that matches no rule falls through to unknown with a safe reply that asks for an order id. That fallback is a feature: later lessons send exactly these messages to the model.

2. Parameterised SQLite queries

Slots never reach SQL as text. Each handler runs one parameterised query against shop.db:

python

def _handle_order_status(self, order_id: str) -> str:
    conn = get_connection(self.db_path)
    cursor = conn.cursor()
    cursor.execute("SELECT id, status, item_name, created_at FROM orders WHERE id = ?", (order_id,))
    row = cursor.fetchone()
    conn.close()
    if row:
        return f"Order {row['id']} containing '{row['item_name']}' is currently {row['status']} (placed on {row['created_at']})."
    return f"Order {order_id} was not found in our system."

The database is seeded with random.Random(42), so ORD-1005 has the same status on your machine as on ours, and the 30-message test set in tests/cases.json gives the same score everywhere.

Trade-Off Analysis: Fast-Path Determinism vs. Semantic Flexibility

Design Dimension	Deterministic Dispatcher (Fast-Path)	LLM Classifier (Slow-Path)
p50 / p99 Latency	< 1 ms / < 3 ms	800 ms / 3,500 ms
Cost per 1M Requests	~$0.02 (Compute/CPU cost only)	~$3.00 – $15.00 (Inference tokens)
Execution Safety	100% deterministic, zero hallucination	Non-deterministic, schema drift risk
Input Flexibility	Rigid (requires matching key phrases/regex)	High (handles typos, complex phrasing)
Failure Mode	Returns `unknown` (Safe Fallback)	Silent hallucination or loop divergence

When to choose differently

Use the Deterministic Dispatcher for high-volume, structured operations where transaction rules are strict (order lookups, account status, policy verification). Switch to the LLM Slow-Path Classifier only when the dispatcher returns intent unknown, indicating high language variance or unmapped user intents.

Production Scale & Latency Math

Let's evaluate the operational impact on a support platform handling 10,000 requests per minute (166 req/sec):

Pure LLM Architecture:
166 req/sec $\times$ 1.2s avg latency = 200 concurrent active connections pinned continuously.
Monthly API cost (at $0.002 per request): $51,840 / month.
Hybrid Fast-Path Architecture (80% Fast-Path Hit Rate):
80% (133 req/sec) served in < 1 ms via SQLite fast-path.
20% (33 req/sec) routed to slow-path LLM.
Pinned concurrent connections drop from 200 to 40.
Monthly API cost drops to $10,368 / month (an 80% direct cost reduction).

What we measured

Running start.sh in the browser sandbox (Ubuntu 24.04, Python 3.12) sends the 30 labelled messages in tests/cases.json to POST /chat:

Metric	Result
Intent accuracy	100% (30 of 30)
Unknown rate	13.3% (the 4 off-topic messages, as intended)
Latency p50 / p99	1.74 ms / 29.56 ms (end to end over HTTP)

The p99 is the first request: it opens the SQLite file and warms Python's caches. Every later request is under 2 ms. Compare that with 800 ms or more for a single model call; this is the budget the rest of the course spends carefully.

Assignment

"Cancel" is currently treated as a refund. Split it into its own intent, cancel_request:

In supportdesk/dispatcher.py, add a CANCEL_KEYWORDS pattern (cancel, cancellation, stop my order) and remove cancel from REFUND_KEYWORDS.
When an order id and a cancel keyword appear together, set the intent to cancel_request. Reply that the order can be cancelled only if its status is placed; otherwise explain why not, using the real status from shop.db.
Add five labelled cancel messages to tests/cases.json and a unit test for a placed and a shipped order.
Run bash start.sh: the tests, the demo and verify.sh must all still pass.

Solution

Solution outline

python

CANCEL_KEYWORDS = re.compile(r"\b(cancel|cancellation|stop my order)\b", re.IGNORECASE)

if order_match:
    order_id = order_match.group(1).upper()
    slots["order_id"] = order_id
    if CANCEL_KEYWORDS.search(normalized_msg):
        intent = "cancel_request"
        reply = self._handle_cancel_request(order_id)
    elif REFUND_KEYWORDS.search(normalized_msg):
        ...

def _handle_cancel_request(self, order_id: str) -> str:
    conn = get_connection(self.db_path)
    row = conn.execute("SELECT id, status FROM orders WHERE id = ?", (order_id,)).fetchone()
    conn.close()
    if not row:
        return f"Order {order_id} was not found."
    if row["status"] == "placed":
        return f"Order {row['id']} has not shipped yet, so it can be cancelled."
    return f"Order {row['id']} is already {row['status']}, so it can't be cancelled. You can request a return instead."

Check the order of the rules: cancel must be tested before refund, or "cancel and refund ORD-1020" would still be a refund.

Say it in an interview

Q1: Why build a deterministic regex/keyword dispatcher when modern LLMs can handle intent classification out of the box?

Answer: Direct LLM classification introduces unacceptable tail latency (p99 > 2 seconds), financial overhead, and non-deterministic security vectors like SQL or prompt injection. A deterministic fast-path handles high-frequency structured queries (80%+ of volume) in under 1 millisecond at near-zero cost, reserving expensive LLMs only for ambiguous, open-ended requests.

Q2: How do you prevent regex-based fast-path routing from mishandling ambiguous queries?

Answer: The fast-path relies on conservative matching with strict slot validation. If required tokens (like a valid order ID schema ORD-\d{4}) are absent or pattern confidence falls below a set threshold, the dispatcher deliberately yields control by returning an unknown intent status. This triggers the slow-path fallback.

Q3: How does direct parameterized SQL execution in the dispatcher protect the serving engine under load?

Answer: By parsing slots deterministically and passing parameters using bound placeholders (?), we eliminate SQL injection vulnerabilities and allow the database engine to reuse compiled query execution plans. This keeps database CPU consumption flat even during high-concurrency request spikes.

Recap

Treat LLM invocations as non-deterministic network RPCs that must be isolated behind deterministic fast-path routers.
The fast-path dispatcher resolves pattern-matched queries in < 2ms using regex slot extraction and parameter-bound SQLite queries.
Failing fast to an unknown intent guarantees a clean handoff to downstream LLM slow-path processors without state corruption.

Challenge (10 minutes)

Modify the dispatcher to measure and enforce a strict execution timeout budget for SQLite queries (e.g., maximum 5ms per database lookup). If the database execution exceeds the budget, log a warning and return an unknown intent fallback response.

Hint 1

Use time.perf_counter() immediately before and after cursor.execute() calls inside query helper functions.

Hint 2

If (end_time - start_time) * 1000 > 5.0, catch the threshold violation and raise or return a fallback dictionary indicating query timeout.

In Day 2, we will build a tool-calling registry and enforce strict schema validation on tool execution, preventing corrupted outputs from polluting downstream workflows.

Symptom	Fix
`Address already in use`	Another app uses port 8090. Run `bash stop.sh`, or `SUPPORTDESK_PORT=8091 bash start.sh`.
`Server failed to pass health check`	Read `supportdesk/.run/app.log`; the Python error is there.
Tests fail after you edit the code	Run `python3 -m unittest discover -s supportdesk/tests` inside the lesson folder to see which one.

Finished the build?

Run verify, then sign in (free) to save your progress and unlock the next lesson. Create a free account →

Day 1: Build a Deterministic Dispatcher

Agenda

State Machine

Flowchart

Component Architecture

The Production Problem: The Cost of Non-Determinism

Architecture of a Deterministic Dispatcher

Control Flow and State Transitions

Key Implementation Snippets

1. The intent rules and slot extraction

2. Parameterised SQLite queries

Trade-Off Analysis: Fast-Path Determinism vs. Semantic Flexibility

When to choose differently

Production Scale & Latency Math

What we measured

Assignment

Solution

Say it in an interview

Recap

Challenge (10 minutes)

Course Navigation

Course Curriculum

The Agent Core

Hardening and Reliability

Observability and Scale

start.sh

stop.sh

Implementation guide — Day 1: Build a Deterministic Dispatcher

Before you start

Run it

Expected output

Try it by hand

Verify and get your completion code

Stop

Troubleshooting

No Demo Video

Resources & Links

📁Repository Structure

GitHub Repository

Questions & Discussion

Leave a Reply Cancel reply

Access Required