Day 6: Write an Idempotent REST Order API with Conditional HTTP Headers — and Verify Cache Invalidation at the Edge

Write an Idempotent REST Order API with Conditional HTTP Headers — and Verify Cache Invalidation at the Edge

In Day 5, we built a low-overhead, rate-limited gRPC server to shield our internal microservices from catastrophic load spikes. However, rate limiting only protects resource capacity; it does not protect state consistency. When a public client encounters a rate-limit drop, a network timeout, or a transient 502 Bad Gateway, its natural reaction is to retry.

If that retried request is a state-changing operation like creating an order or authorizing a payment, a naive retry can result in duplicate orders, double charges, and corrupted state.

Today, we move to the public boundary of our system. We will design and implement a highly resilient REST Order API that guarantees exactly-once processing semantics using custom idempotency keys, protects against lost updates using optimistic concurrency control with standard HTTP conditional headers (ETag and If-Match), and coordinates edge-cache invalidation. This architectural foundation ensures that when clients transition to using the active connection pools we will build in Day 3, they can aggressively retry failed connections without risking data corruption.

The Production Stakes: The Cost of Duplicate Mutations

In 2017, Stripe published an influential analysis of their API idempotency engine. They noted that over lossy mobile connections, up to 12% of API requests are retried due to network drops occurring after the server processed the write but before the client received the acknowledgment. Without idempotency, this results in double charges.

At scale, solving this at the database level is prohibitively expensive. If you rely on database unique constraints alone, you force your storage layer to evaluate duplicate insert attempts, saturating write locks. Furthermore, if the original transaction succeeded but the connection died, simply returning a database error to the client violates the REST contract. The client needs to receive the exact, successful response of the original transaction, even if that transaction occurred minutes ago.

To solve this, we must build an idempotency layer that sits in front of our business logic. This layer intercepts incoming writes, detects duplicates, serializes concurrent identical requests, and serves cached responses for completed transactions.

Additionally, we must protect against the "lost update" problem. If two customer support agents modify the same order concurrently, the last write wins, silently overwriting the previous update. By utilizing RFC 7232 conditional headers (ETag, If-Match, and If-None-Match), we force clients to prove they are mutating the most recent state of a resource.

The Architectural Blueprint

To achieve robust idempotency and safe concurrent mutations, our API gateway and origin server must coordinate across three distinct layers: the Idempotency Guard, the Optimistic Concurrency Validator, and the Edge Cache.

1. The Idempotency Guard

Every mutation request (POST / /orders) must carry an Idempotency-Key header containing a unique UUIDv4 generated by the client. The server tracks these keys using an in-memory or distributed key-value store with two-phase state transitions: