Day 7: Guard Against Overload with Rate Limiting & Resource Isolation — and Witness Backpressure

Day 7: Guard Against Overload with Rate Limiting & Resource Isolation — and Witness Backpressure

In Day 6, we instrumented our API with metrics and tracing to pinpoint a latency hotspot in our database query layer. But tracking latency is reactive; it tells you your system is dying, but it does not save it. Today, we shift from observation to active defense. We will build and break an active overload protection system that prevents our service from crashing when traffic spikes beyond its physical capacity.

The Continuity Contract

• Where we are coming from: In Day 6, you built a Prometheus and OpenTelemetry instrumentation layer that exposed how slow database queries degrade upstream API performance.

• Where we are going: Today, we use those metrics to trigger backpressure. We will implement token-bucket rate limiting and active concurrency limits. This hardened server will serve as the baseline for Day 8, where we will execute a full-scale distributed load test and analyze cascading failures under multi-region stress.

The Production Stakes: The Silent Killer of Node.js Services

In single-threaded runtimes like Node.js, the event loop is your most precious resource. When requests arrive faster than your downstream dependencies (like PostgreSQL or an external payment gateway) can process them, Node.js does not stop accepting connections. Instead, it continues to queue incoming HTTP requests in memory.

This behavior is a classic backpressure failure. If each buffered request retains 10 KB of metadata, closures, and database client allocations, a sustained spike of 100,000 pending requests will consume 1 GB of heap memory. This quickly triggers the V8 engine’s garbage collector to run continuously, consuming 100% CPU, before finally crashing the process with an Out Of Memory (OOM) error.

This is exactly what happened during Stripe's early API outages. Their upstream servers accepted connections faster than their backend databases could process them, leading to worker exhaustion and cascading failures across their entire API fleet. They resolved this not just by adding database capacity, but by introducing aggressive, layered rate limiting and concurrency throttling at the edge and within the service mesh.

The Intuition: The Nightclub Bouncer and the Fire Marshal

To protect a system from overload, you need two distinct defensive layers: