Day 5: Replicate and read your own writes

Shards spread writes. Most traffic, though, is reads, and the classic way to scale reads is to keep copies of the data — replicas — and read from them. Replication also keeps you alive when a database machine dies. It comes with one famous bug: you place an order, open "my orders", and the order isn't there. In this lesson you build leader–follower replication yourself, reproduce that bug 100 times out of 100, and fix it with read-your-own-writes.

The problem

Every order shard from lesson 4 becomes a leader with a follower. All writes go to the leader. Reads go to the follower, which takes load off the leader and could live in another data centre. The follower is updated asynchronously: the leader doesn't wait for it. That keeps writes fast, but it means the follower is always a little behind. The gap is replication lag.

What you build

Real databases replicate by shipping their write-ahead log. You build the same idea with three small changes:

• A changelog per shard. In shards.py, insert_order() writes the order and a row in a changelog table in one transaction. The changelog row gets an auto-incrementing LSN (log sequence number) — the write's position in the history. insert_order() returns it.

• A follower per shard. orders-shardN-follower.db has the same orders table plus replica_state.applied_lsn: how far it has caught up.

• replica.py. Every 0.5 seconds, for each shard, it reads changelog rows with lsn > applied_lsn, applies them to the follower and advances applied_lsn — in one follower transaction, so a crash can't apply half a batch. It also records lag: the time from the leader's commit to the follower's apply.

Then the read path in orders_for(user_id, min_lsn):

• READ_MODE=follower: always read the follower.

• READ_MODE=ryw (read-your-own-writes): if the caller says "I wrote something at LSN 812" and the follower's applied_lsn is below 812, read the leader instead. Otherwise read the follower.

The API carries the LSN to the client: POST /orders returns {"order_id": …, "shard": …, "lsn": 812} and the client asks for GET /users/<id>/orders?after=812. In a web app the LSN would live in the session or a cookie for a few seconds after a write.

Break it

stale.py does what a shopper does: place an order, then immediately open "my orders". It repeats this 100 times with fresh users.

With READ_MODE=follower: