Day 4: Catch a memory leak and an exhausted pool

Debug Microservices with OpenTelemetry and Prometheus ~5 min read 40 min hands-on

Leak database connections and memory on purpose, and catch both with cause alerts before (or without) customers noticing.

Some failures don't arrive as a spike. A connection pool loses one connection every few hundred requests; memory grows a megabyte a second. Everything looks fine until, suddenly, nothing works. Today you inject both of these slow-burning faults into ShopStream and catch them with cause alerts β€” metrics that warn you before, or right as, customers start to fail.

What you build

  • A database connection pool in inventory: 5 connections (a BoundedSemaphore standing in for real ones). Each /reserve acquires a connection with a 1-second timeout; if none is free, inventory returns 503 "connection pool exhausted". The wait is a span, db_pool_acquire, so it shows up in traces.

  • Pool metrics: db_pool_in_use, db_pool_size and the histogram db_pool_wait_seconds.

  • Two new faults via POST /admin/fault:

  • pool_leak_rate β€” the fraction of inventory requests that never return their connection (a missing finally).

  • leak_kb β€” kilobytes checkout keeps forever per request (a cache with no eviction).

  • Two cause alerts:

    yaml

The leak in code

In real code this is a return or an exception between acquire and release. The fix is always the same: release in a finally block, or use a context manager.

What we measured

Steady traffic at 20 checkouts per second, pool of 5:

Pool leak, 5% of requests:

  • First failed checkout after 4 s β€” 5 connections at 5% leak rate is about 100 requests, and the leak plus the 1 s acquire timeout turns into failures quickly.

  • InventoryPoolSaturated fired after 19 s; by then 64 of 113 checkouts had failed and the pool was 100% in use.

Memory leak, 64 KB per request (about 1.3 MB/s):

  • Checkout's resident memory grew from 51 MB to 116 MB.

  • CheckoutMemoryLeak fired after 50 s. None of 1,004 checkouts failed β€” a memory leak is invisible to customers until the process is killed for running out of memory, which is exactly why you alert on the trend.

Read the full lesson free

This lesson is one of the first 3 in its course, free with an account β€” the full article, implementation guide and scripts. No card needed.

Already a member? Sign in