Every system design answer starts with a claim like "this will handle the load". Interviewers push on that claim, and the honest answer is usually "I don't know, I never measured it". In this lesson you measure it. You run ShopStream, a small online-shop API, on your own machine, aim a load generator at it, and watch it go from fast to slow to broken. By the end you'll know three numbers for your system — throughput, median latency and p99 latency — and which one breaks first.
The problem
ShopStream sells 2,000 products. Each product page shows the product, its price, its stock and its star rating. The rating is computed from 200,000 reviews every time someone opens the page. That's a realistic mistake: the query is correct, it's fast enough on a developer's laptop with one user, and nobody notices until a sale brings real traffic.
The question for this lesson: how many shoppers can ShopStream serve before pages get slow, and what does "slow" look like?
What you build
Run start.sh and it writes three files into ./shopstream:
db.pycreatesshop.db(SQLite in WAL mode) withproducts,reviewsandorders, filled with seeded random data so your numbers are comparable with ours. Thereviewstable has no index onproduct_id— on purpose.product_page()runsCOUNT(*)andAVG(stars)over it, so every page view scans all 200,000 rows.app.pyis the API, built on Python's standard-libraryThreadingHTTPServer: one thread per connection, one SQLite connection per thread. Routes:GET /health,GET /products/<id>,POST /orders.load.pyis a tiny load generator. It starts N threads ("users"); each keeps a connection open and requests random product pages from a hot set of 200 as fast as the server answers, for a few seconds. It reports requests per second, p50 and p99 latency, and errors.
There is no pip install anywhere in this course. Everything runs on Python 3.10+ and, from lesson 2, a local Redis. That's deliberate: you can read every line that matters.
The order endpoint already does one thing right. It decrements stock with a single conditional statement:
If the row count is 0, the product is sold out and the API returns 409. Two buyers racing for the last unit can't both win, because the check and the write are one statement. Remember this; it's why the cache in lesson 2 can't cause overselling.
Break it
start.sh runs the load test at 1, 5, 10, 25 and 50 concurrent users, then at 200. Here's what we measured on a 2-core machine:
Your numbers will differ with your CPU. The shape won't.
Measure — and read the numbers like an engineer
Throughput hits a ceiling almost immediately. One user gets 149 requests per second; five users get 300; after that it's flat. The server has two cores and each page costs about 6 ms of CPU for the review scan. Two cores × (1000 ms ÷ 6 ms) ≈ 330 pages per second. That's the ceiling, and no number of extra users moves it.
Latency is what grows. Once the CPU is busy, every extra user just waits in line. This is Little's law: users in the system = throughput × time in the system. With throughput stuck at about 300 per second, 50 users means each request spends about 50 ÷ 300 ≈ 167 ms in the system. We measured a median of 164 ms. The law works.
p99 is where users feel it. At 50 users the median is 164 ms but the slowest 1% take over half a second. At 200 users the slowest take more than 4 seconds. On a real page that loads 20 resources, almost every visitor hits at least one p99 request. That's why system design conversations talk about p99, not averages.
Two bugs we hit on the way (and fixed)
Building this lesson, the first version of the load test showed a strange, constant 44 ms per request even with one user. That was Nagle's algorithm on the server colliding with delayed ACKs on the client: small responses were held back waiting for an acknowledgement that was itself being delayed. One line fixes it — TCP_NODELAY on every accepted socket, in Handler.setup(). Latency fell from 44 ms to 6.5 ms.
The second: at 200 users, 106–161 requests failed with connection resets. ThreadingHTTPServer listens with a backlog of 5 by default. When 200 clients connect at once, the kernel's queue of not-yet-accepted connections overflows and new ones are dropped or retried after a second. Setting request_queue_size = 128 brought errors to zero.
Neither bug is about Python. Every server has a listen backlog and every TCP stack has Nagle. Knowing they exist is worth a lot in an interview when someone asks "why is the latency floor 40 ms?"
Say it in an interview
Q: Your service handles 300 requests per second. Product wants 3,000. What do you do?
First find the bottleneck. Here it's CPU on the review aggregation — about 6 ms per page. Options in order of cost: stop doing the work per request (cache the page or precompute the rating when a review is written), make the work cheaper (an index or a summary table), then add capacity (more app instances, but the database becomes the next bottleneck). Precomputing gives the biggest jump for the least effort.
Q: Why report p99 and not average latency?
Averages hide the tail. Our median at 50 users was 164 ms but p99 was 514 ms. Users who make many requests — or pages that make many calls — hit the tail almost every time. p99 tells you what your unhappiest regular users see.
Q: Throughput stopped growing but latency kept rising. Why?
Because the system was saturated. Once the CPU is at 100%, extra concurrent requests can't be processed any faster; they wait in a queue. By Little's law, latency = concurrency ÷ throughput, so doubling users at fixed throughput doubles latency.
Recap
Throughput has a ceiling set by your bottleneck; here, two cores ÷ 6 ms of CPU per page ≈ 300 pages per second.
Past that ceiling, more users only add queueing. Latency grows linearly (Little's law) and p99 grows faster.
Measure p50 and p99 together, and look for errors. Ours came from a listen backlog of 5.
Small infrastructure defaults — Nagle, backlog — can dominate your numbers. Measure before you believe.
Challenge (10 minutes)
Make the product page fast without a cache: keep the rating correct, but stop scanning 200,000 rows per request. Then run python3 load.py --users 50 --seconds 4 and compare with the table above.
Hint 1
An index on reviews(product_id) lets SQLite read only the ~100 reviews for one product. Add it in db.seed() with CREATE INDEX IF NOT EXISTS.
Hint 2
Even faster: keep review_count and rating_sum columns on products and update them when a review is inserted. The page then reads one row. This is "precompute on write" — the same trade-off a cache makes, but kept exactly in sync.
What you should see
With the index we measured 2,430 requests per second at 50 users (up from 280) and p99 55 ms (down from 514). The summary columns go further still. Don't keep it for lesson 2 — the next lesson needs the slow page to show what a cache does.
Next
In lesson 2 you put Redis in front of the product page. Throughput goes from 280 to over 5,000 requests per second — and then you'll cause a thundering herd on purpose and fix it.