Day 4: Refine the API with Idempotent Operations — and Avoid Double Writes

Day 4: Refine the API with Idempotent Operations — and Avoid Double Writes

In the previous lesson, we built a write-ahead log (WAL) to ensure that once our storage engine acknowledges a write, that write survives unexpected process crashes. Today, we tackle a failure mode that occurs before a write can even be safely committed: the network retry storm.

In any real-world distributed system, the network is an untrusted, asynchronous medium. Packets are delayed, duplicated, or dropped entirely. When a client sends a write request to your key-value store, and the network drops the connection before the server's acknowledgment reaches the client, the client is left in a state of uncertainty. Did the server crash before processing the write? Or did it write the data successfully and fail only while sending the response?

To guarantee safety, the client must retry. But if the write was already processed, a naive retry results in a double write. If the operation is a simple value replacement, a double write might seem harmless. But if the operation is an increment, an append, or a conditional update, a double write corrupts your application state.

Today, we will design and implement an idempotency layer for DurableKV that guarantees "exactly-once" execution semantics (implemented as at-least-once delivery plus idempotent processing) even under severe packet loss and client retries.

The Production Stakes: The $10 Million Retry Storm

To understand why this matters, look at the public post-mortems of major payment and infrastructure platforms. In 2020, a major fintech provider experienced a cascade of duplicate charges totaling millions of dollars. The root cause? A database failover caused transient latency. Clients timed out waiting for responses and retried their requests. Because the API endpoints lacked a robust, persistent idempotency layer, the retried requests were treated as brand-new transactions, executing the debit logic a second time.

A similar failure mode exists in storage systems. If a client attempts to append data to a log-structured key-value store:

Without idempotency, your data integrity depends entirely on a perfect network—which does not exist.

System Architecture & Control Flow

To prevent double writes, we introduce an Idempotency Filter at the entry point of our API.

Component Architecture

The system consists of three main components:

1. API Gateway / Router: Receives incoming HTTP requests, extracts the Idempotency-Key header, and coordinates with the Idempotency Registry.

2. Idempotency Registry: An in-memory, thread-safe cache backed by a persistent journal that tracks the state of every active transaction token.

3. Storage Engine (DurableKV): The underlying key-value engine that executes the state changes once they are cleared by the registry.

Refer to the component layout below:

The Request Lifecycle

When a write request arrives with an Idempotency-Key: