Day 4: Catch a memory leak and an exhausted pool
Leak database connections and memory on purpose, and catch both with cause alerts before (or without) customers noticing.
Some failures don't arrive as a spike. A connection pool loses one connection every few hundred requests; memory grows a megabyte a second. Everything looks fine until, suddenly, nothing works. Today you inject both of these slow-burning faults into ShopStream and catch them with cause alerts β metrics that warn you before, or right as, customers start to fail.
What you build
A database connection pool in inventory: 5 connections (a
BoundedSemaphorestanding in for real ones). Each/reserveacquires a connection with a 1-second timeout; if none is free, inventory returns 503 "connection pool exhausted". The wait is a span,db_pool_acquire, so it shows up in traces.Pool metrics:
db_pool_in_use,db_pool_sizeand the histogramdb_pool_wait_seconds.Two new faults via
POST /admin/fault:pool_leak_rateβ the fraction of inventory requests that never return their connection (a missingfinally).leak_kbβ kilobytes checkout keeps forever per request (a cache with no eviction).Two cause alerts:
The leak in code
In real code this is a return or an exception between acquire and release. The fix is always the same: release in a finally block, or use a context manager.
What we measured
Steady traffic at 20 checkouts per second, pool of 5:
Pool leak, 5% of requests:
First failed checkout after 4 s β 5 connections at 5% leak rate is about 100 requests, and the leak plus the 1 s acquire timeout turns into failures quickly.
InventoryPoolSaturatedfired after 19 s; by then 64 of 113 checkouts had failed and the pool was 100% in use.
Memory leak, 64 KB per request (about 1.3 MB/s):
Checkout's resident memory grew from 51 MB to 116 MB.
CheckoutMemoryLeakfired after 50 s. None of 1,004 checkouts failed β a memory leak is invisible to customers until the process is killed for running out of memory, which is exactly why you alert on the trend.
Read the full lesson free
This lesson is one of the first 3 in its course, free with an account β the full article, implementation guide and scripts. No card needed.
Already a member? Sign in