Day 23: Implement Snapshotting and Compacting the Write-Ahead Log — and Limit Disk Usage and Recovery Time
Day 23: Implement Snapshotting and Compacting the Write-Ahead Log
In our previous lesson, we built a Write-Ahead Log (WAL) and an in-memory MemTable. Every write was appended to an on-disk log before updating our memory state. This guaranteed durability: if our process crashed, we could reconstruct the entire state by replaying the log from start to finish.
However, our engine has a fatal architectural flaw: the WAL grows indefinitely.
Today, we will implement Snapshotting and WAL Compaction. By the end of this lesson, your engine will be able to freeze its in-memory state, flush it to a highly compressed snapshot file, and safely truncate (compact) the obsolete WAL segments. This bounds both disk space usage and database recovery time to a predictable constant, regardless of how many millions of writes have occurred.
The Production Stakes: The Threat of Unbounded Replay
In production, ignoring WAL compaction is a guaranteed recipe for a high-severity incident.
Consider a service sustaining 10,000 updates per second to a set of 1,000 hot keys (e.g., real-time inventory balances or session states). Each mutation appends 256 bytes to the WAL.
In 1 hour: The WAL grows to $10,000 times 256 text{ bytes} times 3,600 text{ seconds} approx 9.2 text{ GB}$.
In 24 hours: The WAL reaches 220 GB, even though the active dataset size (the 1,000 keys) is less than 1 megabyte!
If this node crashes, the recovery process must read and parse 220 GB of historical junk from disk, replaying millions of obsolete states just to reconstruct a 1 MB map. Your recovery time (Mean Time to Recover, or MTTR) stretches from milliseconds to hours. This is not theoretical: early deployments of Apache Cassandra and custom RocksDB-based storage engines frequently suffered from "recovery loops," where a node crashed due to memory pressure, and then failed to join the cluster for hours because it was stuck replaying years of uncompacted logs.
System Architecture and Data Flow
To solve this, we introduce a dual-file storage layout:
Snapshots (
snapshot.<timestamp>.db): A serialized representation of the MemTable at a specific logical moment.Active WAL (
wal.<timestamp>.log): An append-only log that only contains writes that occurred after the latest snapshot was taken.
When we trigger a snapshot:
Keep going with the full lesson
The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.
Already a member? Sign in