Day 27: Perform Capacity Planning: Benchmark Throughput and Latency Under Realistic Load — and Find the Breaking Point

Day 27: Perform Capacity Planning: Benchmark Throughput and Latency Under Realistic Load — and Find the Breaking Point

In the previous lesson, we completed our core storage engine, implementing a Write-Ahead Log (WAL) and an in-memory Memtable to guarantee durability while maintaining low write latencies. Our engine works correctly under normal operations. However, we do not yet know its limits. We have no objective data on how many writes per second it can sustain before latency degrades, what its primary bottleneck is, or how it behaves when pushed beyond its saturation point.

Today, we will build a high-fidelity benchmarking suite designed to expose the true capacity of DurableKV. We will explore why standard industry benchmarks often lie to us, dive deep into the phenomenon of Coordinated Omission, and implement an open-loop load generator that reveals the catastrophic tail latencies hidden by traditional closed-loop testing tools.

The Problem: The "Works on My Machine" Performance Trap

Most engineers evaluate the performance of their storage systems using a simple loop:

This is a closed-loop client. It sends a request, blocks until it receives a response, and only then sends the next request.

If the database experiences a temporary stall—such as a 100-millisecond disk sync pause during a WAL rotation—the client blocks and waits. Because the client is blocked, it stops sending requests during the stall. It "coordinates" its request rate with the server's omission to respond quickly.

This coordination introduces a massive measurement bias known as Coordinated Omission. If a system stalls for 10 seconds, a closed-loop benchmark might record only a single request taking 10 seconds. Meanwhile, thousands of virtual requests that would have arrived in a real-world, production environment were never sent. Your benchmark reports a healthy 99th percentile latency of 5 milliseconds, while in production, your users experience timeouts and cascading failures.

The Intuition: The Bus Stop Analogy

Imagine a bus route where a bus is scheduled to depart every 10 minutes.

Suppose the 10:10, 10:20, and 10:30 buses are canceled due to an engine breakdown. The next bus finally arrives at 10:40.

If you measure the delay from the bus's perspective (closed-loop):

• The bus at 10:00 took 0 minutes to depart.

• The bus at 10:40 took 0 minutes to depart (it left immediately once turned on).

• Average delay reported: 0 minutes.

If you measure the delay from the passengers' perspective (open-loop):

• Passengers who arrived at 10:10 waited 30 minutes.

• Passengers who arrived at 10:20 waited 20 minutes.

• Passengers who arrived at 10:30 waited 10 minutes.

• Average passenger wait time: 20 minutes.

Production systems behave like the bus stop. Your users do not coordinate with your server's stalls; they keep clicking, purchasing, and searching. To find the true breaking point of DurableKV, we must build an open-loop load generator that schedules requests at a strict, independent rate, measuring latency from the intended schedule time rather than the actual dispatch time.

System Architecture: Open-Loop vs. Closed-Loop

Our benchmarking suite splits the workload generator into two distinct execution paths to demonstrate this bias:

1. The Closed-Loop Client (Worker-Bound)