Day 14: Build a Client-Side Load Balancer with Retries and Timeouts — and Survive Node Failures Gracefully

Day 14: Build a Client-Side Load Balancer with Retries and Timeouts

In our previous work, we built a replicated storage node cluster capable of persisting keys and values to disk. However, our client was primitive: it talked to a single hardcoded node address. If that node stalled, crashed, or experienced a transient network hiccup, our entire application came to a grinding halt.

Today, we transition from a fragile single-point connection to a resilient, client-side load balancer. In Day 15, we will introduce dynamic node discovery via a gossip protocol. Today, we lay the critical data-path foundation: how a client distributes load across a pool of nodes, detects failures, times out quickly, and retries requests safely without triggering a self-inflicted system collapse.

The Production Stakes: The Threat of the Retry Storm

When a downstream service slows down, the natural human instinct is to try again. In distributed systems, this instinct is lethal.

On March 12, 2015, AWS DynamoDB experienced a localized high latency event. Clients, configured with aggressive, naive retry loops, immediately began retrying failed requests. Because these clients retried instantly and repeatedly, the total request volume hitting the storage engines amplified by orders of magnitude. This "retry storm" saturated network links and CPU queues, turning a minor, transient hardware glitch into a prolonged multi-hour outage. The recovering nodes were immediately crushed by a wall of queued retries the moment they came back online.

To survive in production, your client-side load balancer must adhere to three strict rules:

1. Timeouts must be bounded and aggressive: Never wait indefinitely for a connection or a response.

2. Backoff must be exponential and jittered: Space out your retries, and randomize them to break up synchronization waves (the "thundering herd" problem).

3. Retries must target different nodes: If Node A is failing, retrying on Node A is foolish. Retrying on Node B increases the probability of success.

Component Architecture

The client-side load balancer sits entirely within the client library process. It intercepts every read (Get) and write (Put) request before it hits the network.

The load balancer maintains an internal list of known node endpoints. When a request is initiated:

1. Select: It uses a Round-Robin selection algorithm to pick the next candidate node.

2. Execute: It dispatches the HTTP request inside a execution context governed by a strict timeout.

3. Evaluate: If the request succeeds, the result is returned to the application. If it fails (connection refused, HTTP 5xx, or timeout), the load balancer marks that node as temporarily unhealthy, calculates a randomized backoff delay, selects the next node in the pool, and tries again.

Core Concepts & Trade-Offs

1. Round-Robin vs. Smart Routing