Day 15: Partition the Network Between Replicas — and Observe Stale Reads

Day 15: Partition the Network Between Replicas — and Observe Stale Reads

In Day 14, we built an asynchronous replication engine that allowed our leader node to propagate writes to a follower replica in the background. It gave us low-latency writes and read scalability. Today, we are going to break it.

We will introduce a simulated network partition—the classic "split-brain" precursor—and watch our system violate linearizability. You will write the code to partition your local cluster, execute a write to the leader, read from the partitioned replica, and directly observe a stale read. This exercise is the practical realization of the PACELC theorem: when a partition occurs ($P$), our system chooses latency/availability ($A$) over consistency ($C$), leading to stale reads.

The Production Stakes: Why This Outage Will Wake You Up

In 2013, Kyle Kingsbury (creator of the Jepsen testing suite) published an analysis of MongoDB's default replication behavior. Under a network partition, isolated secondaries continued to accept read requests with the default ReadPreference.secondary configuration. Because these secondaries were cut off from the primary, they had no way of knowing that the primary had already accepted new writes. Clients reading from these secondaries were served stale, out-of-date records.

In high-throughput environments, this is not just an academic concern. Imagine a financial ledger where a user transfers balance out of an account on the leader, but a subsequent authorization check reads from a partitioned replica. The replica, unaware of the transfer, shows the old balance, allowing a double-spend.

If you allow reads from replicas without explicit coordination, you are trading correctness for read scalability. Today, we will measure the exact cost of that trade.

The Intuition: The Photocopy Room

Imagine a busy office. The Manager (Leader) sits in the main office, taking incoming orders and writing them in a master ledger. In the back room, a Clerk (Replica) maintains a photocopy of the ledger. Whenever the Manager writes a new order, they send a runner down the hallway to tell the Clerk to update their copy.

Customers who want to read orders can stand in line at the Manager’s window or walk around to the Clerk’s back window.

Now, imagine a fire door accidentally shuts and locks in the hallway (a network partition). The runner can no longer reach the back room.

1. A customer walks up to the Manager and places a new order: "Order 105: 50 Units". The Manager writes it down.

2. Simultaneously, another customer walks up to the Clerk's back window and asks: "What is the latest order?"

3. The Clerk, completely unaware that the hallway is blocked or that a new order was placed, looks at their isolated ledger and says: "Order 104: 12 Units".

This is a stale read. The Clerk did not fail; they answered quickly and politely. But they lied.

System Architecture & Data Flow

Our system consists of two nodes running as independent HTTP servers on different ports:

• Node 8001 (Leader): Accepts writes (POST /set) and asynchronously replicates them to Node 8002.

• Node 8002 (Replica): Accepts replication requests (POST /replicate) and serves reads (GET /get).

To simulate a network partition without requiring root privileges or complex virtual networks, we implement an application-level partition table on each node. By calling /admin/block?peer=8002 on Node 8001, we instruct Node 8001's outgoing client to drop any network requests destined for Node 8002.

Core Mechanics: The Partitioned Replication Loop

To understand how stale reads manifest in code, look at how the leader handles replication failures when a peer is blocked. In a real system, we cannot let replication block the main write path indefinitely. We use a non-blocking background thread or a short socket timeout.