Day 10: Kill a Replica Mid-Write — and Observe Divergent State

Day 10: Kill a Replica Mid-Write — and Observe Divergent State

In Day 9, we built our first multi-replica storage coordinator. It accepted write requests and replicated them to three storage nodes in parallel. Under ideal conditions, this layout provides fault tolerance. However, in production, ideal conditions are short-lived.

Today, we will expose the structural flaw in our naive replication engine. We will write a key, kill a replica mid-write, and observe a critical distributed systems failure mode: divergent state.

The Production Stakes: The Cost of Partial Writes

When a multi-replica write operation fails halfway through, what happens to the replicas that already processed the write?

In a naive system, those writes remain applied. The coordinator might return an error to the client, but the system's state has already mutated unevenly. This is not a theoretical edge case; it is a primary driver of silent data corruption in production.

In 2018, a major financial platform experienced a database divergence incident during a routine network re-routing. The database coordinator attempted to write an account balance update to three replicas. One replica became unreachable mid-write due to a transient network drop. The coordinator returned a write failure to the upstream service.

However, two of the replicas had already updated their local state. The upstream service, seeing the write failure, retried the operation on a different coordinator node. Due to an read-repair misconfiguration, subsequent read requests routed to the stale replica served the old balance, while writes to the other replicas advanced the new balance. This split-brain behavior resulted in thousands of inconsistent account audits before the nodes could be manually resynchronized.

This class of bug is why systems like Apache Cassandra, Amazon Dynamo, and CockroachDB invest heavily in consensus protocols, Paxos/Raft log replication, or strict Quorum read/write mechanics ($R + W > N$). Without these safety nets, a single network hiccup or node crash mid-write results in permanent, silent divergence.

The Core Concept: Naive Parallel Replication vs. Atomic Consensus

In our current implementation, the coordinator employs a fire-and-forget parallel write strategy. It dispatches HTTP POST requests to all three replicas. If any replica returns an error or times out, the coordinator returns an error to the client.