Day 22: Build a Coordination Service for Cluster-Wide Configuration Changes — and Prevent Split-Brain Scenarios

Day 22: Build a Coordination Service for Cluster-Wide Configuration Changes — and Prevent Split-Brain Scenarios

In Day 21, we built a durable local storage engine using a Write-Ahead Log (WAL) and LSM-Tree to ensure a single node survives crashes without losing data. But a single node is a single point of failure. To survive datacenter outages, we must scale horizontally. The moment we introduce multiple nodes, however, we face the most terrifying failure mode in distributed systems: split-brain.

Today, we will build a quorum-backed Coordination Service using Single-Decree Paxos to manage cluster-wide configuration changes. This service acts as the single source of truth for which nodes are active and which node is the designated primary. In Day 23, we will expand this replication model to build a fully replicated state machine using Raft, allowing the storage engine itself to handle active-active replication.

The Production Stakes: The GitHub 2018 Outage

On October 21, 2018, GitHub went offline for over 24 hours. The root cause? A routine network partition.

A brief connectivity loss between their primary US East Coast datacenter and their US West Coast backup site caused their automated database orchestrator to failover. The orchestrator promoted a West Coast replica to be the primary. However, the East Coast primary was still running and accepting writes from a subset of clients that could still reach it.

This is split-brain: two nodes simultaneously believing they are the sole primary, both accepting conflicting writes. Reconciling divergent databases is a manual, error-prone nightmare that often leads to permanent data loss.

To prevent this, production engines like Spanner, Aurora, and CockroachDB never rely on a single node's perspective to make topology changes. They use a quorum-based coordination service (like Chubby, ZooKeeper, or custom Paxos groups). A topology change can only be committed if a strict majority of the coordination nodes agree.

The Core Concept: Quorum-Based Consensus

To safely change cluster configuration (such as promoting a new primary), we must guarantee that only one configuration update can succeed at any given version or epoch. We achieve this using Single-Decree Paxos.

The protocol operates in two phases across $N$ coordinator nodes: