Day 24: Saturate a Node’s CPU/Disk/Network — and Watch Latency Spike

On a quiet Tuesday morning, a single slow disk in a storage cluster can bring down an entire multi-region payment gateway.

This is not a hypothetical scenario. In 2015, Amazon DynamoDB experienced a catastrophic five-hour outage in the US-East-1 region. A high-volume metadata service saturated the storage nodes hosting partition tables. Because the storage nodes lacked strict admission control, they accepted every incoming request, queueing them up indefinitely. As queues grew, request latencies swelled from milliseconds to tens of seconds. Upstream services timed out, assumed the nodes were dead, and retried. This retry storm acted as a self-inflicted Distributed Denial of Service (DDoS) attack, cementing a total system collapse.

In this lesson, we will build, analyze, and deliberately break a storage engine node under simulated disk saturation. We will study the mechanics of queueing delay, understand why unbounded queues are a production hazard, and implement a bounded-queue admission controller that uses early rejection (load shedding) to protect the node's p99 latency.

The Core Concept: Little's Law & Head-of-Line Blocking

To understand why resource saturation spikes latency, we must look at the mathematical engine of queueing theory: Little's Law.

$$L = lambda cdot W$$

Where:

• $L$ is the average number of requests inside the system (both being processed and waiting in queue).

• $lambda$ is the arrival rate of incoming requests.

• $W$ is the average time a request spends in the system (latency).

If your storage disk can only sync 100 writes per second ($mu = 100$), and your arrival rate ($lambda$) spikes to 200 requests per second, your system is oversaturated.

If your queue is unbounded, the number of waiting requests ($L$) will grow linearly toward infinity. Consequently, the average waiting time ($W$) also approaches infinity. Every new request must wait behind every previously queued request—a phenomenon known as Head-of-Line (HoL) blocking.

By placing a Bounded Queue in front of our worker pool, we enforce a strict upper bound on $L$. If the queue capacity is $C$, and we have $M$ active workers each taking $T$ seconds to process a write, the maximum time any accepted request can wait in the queue is:

$$W_{max} approx frac{C cdot T}{M}$$

Any request arriving when the queue is full is instantly rejected with an HTTP 503 Service Unavailable status. This is load shedding. It preserves the latency budget of the requests we do accept, preventing a slow disk from turning into a dead node.