Day 6: Establish Inter-Node Communication with gRPC — and Handle Network Glitches

Day 6: Establish Inter-Node Communication with gRPC — and Handle Network Glitches

In the previous design phase, we built a robust, single-node storage engine capable of writing SSTables and managing a Write-Ahead Log (WAL) to survive local process crashes. However, a single node is a single point of failure. To achieve high availability and scale queries, our key-value store must evolve into a distributed system.

Today, we transition from disk-bound code to network-bound code. We will build the inter-node communication layer using gRPC. More importantly, we will address the reality of the network: the network is a lie. It drops packets, injects latency, and partitions nodes.

By the end of this lesson, you will have implemented a resilient gRPC client-server transport layer capable of surviving transient network partitions, server restarts, and high-latency spikes using client-side deadlines, exponential backoff, and full jitter.

The Production Stakes: The Self-Inflicted DDoS (Retry Storms)

On September 20, 2015, Amazon Web Services (AWS) experienced a major outage in its DynamoDB service in the US-East-1 region. The root cause was a metadata service overload. When the storage nodes detected transient network glitches, they began retrying their requests to the metadata service.

Because these clients used naive retry logic—retrying immediately or at fixed intervals without randomized delays—they synchronized. This synchronization created a massive, periodic traffic spike known as a thundering herd or retry storm. The metadata service was hit with a self-inflicted Distributed Denial of Service (DDoS) attack. Every time the metadata service tried to recover, the synchronized wave of client retries immediately knocked it offline again.

To prevent this in our system, we must implement three core patterns:

1. Deadlines (Timeouts): Prevent client threads from hanging indefinitely on slow network calls.

2. Exponential Backoff: Increase the delay between successive retries to give a struggling server breathing room.

3. Full Jitter: Randomize the retry delays to break up synchronization and spread the load evenly over time.

Component Architecture & System Fit

The network layer sits directly above our storage engine. In a multi-node cluster, a write request arriving at Node A may need to be forwarded to Node B (the partition owner or partition replica).

The gRPC server exposes two primary RPCs: Get and Put. When a peer node makes a call, the request passes through our custom client-side gRPC Unary Interceptor. This interceptor intercepts the call, wraps it in a retry loop, applies a deadline, and calculates randomized backoff times if a transient failure occurs.

Why gRPC? (Trade-off Honesty)

We chose gRPC over HTTP/JSON or custom raw TCP sockets. Let's look at the trade-offs:

Core Mechanics: Exponential Backoff and Full Jitter