Day 11: Surviving the Avalanche — Load Shedding and Circuit Breaking.

Day 11: Surviving the Avalanche — Load Shedding and Circuit Breaking

Yesterday, you built a version-aware REST API that dynamically routes requests to appropriate controllers without modifying the URL path. It solved the problem of API evolution. But today, we face a far more brutal reality: the network is hostile, downstream dependencies fail, and traffic is not a smooth stream—it is a series of spikes that can easily crush your services.

Today, you will build a high-performance API Gateway layer that sits in front of your version-aware router. This layer is designed with one goal: ensure the system survives a 10x traffic spike by shedding load, failing fast, and degrading gracefully.

1. The Production Stakes: The Anatomy of a Cascading Failure

In October 2021, Roblox suffered a historic 73-hour outage. While the root cause was a low-level database performance issue, the prolonged recovery was driven by a classic distributed systems phenomenon: the retry storm. When the core systems slowed down, upstream services and client applications repeatedly retried failed requests. This multiplied the traffic load by orders of magnitude, preventing the database from ever recovering.

To understand why this happens, let’s look at the capacity math of a typical gateway thread pool or concurrency queue.

The Math of Thread Starvation

Imagine your API Gateway allocates a pool of $W = 100$ worker threads (or concurrent execution slots in an asynchronous event loop) to handle incoming requests for an order processing service.

Under normal conditions:

• Downstream Latency ($L_{normal}$): $50text{ ms}$ ($0.05text{ s}$)

• Max Throughput per Worker: $1 / 0.05 = 20text{ req/sec}$

• Total Gateway Capacity: $100 times 20 = 2,000text{ req/sec}$

Now, suppose the downstream database experiences lock contention, and latency spikes:

• Degraded Latency ($L_{degraded}$): $5,000text{ ms}$ ($5text{ s}$)

• Max Throughput per Worker: $1 / 5 = 0.2text{ req/sec}$

• Total Gateway Capacity: $100 times 0.2 = 20text{ req/sec}$

If your system is receiving $1,000text{ req/sec}$, the gateway will exhaust all 100 worker threads in exactly 100 milliseconds. Once the thread pool is saturated, the gateway cannot accept any new requests—even for unrelated, perfectly healthy services (like your static homepage or user profile endpoints). The failure of one downstream service has cascaded to bring down the entire system.

To prevent this, we must implement two defensive patterns:

1. Token-Bucket Load Shedding: Reject excess traffic at the edge in $< 1text{ ms}$ before it can consume worker threads.

2. Stateful Circuit Breaking: Detect when a downstream service is failing or slow, trip a circuit breaker to stop sending traffic to it, and instantly return a graceful, degraded fallback response.

2. The Core Mechanisms

We can visualize these defenses as a water filtration and bypass system.

Token-Bucket Rate Limiting (Load Shedding)