Hands-On System Design Tutorial

# Day 27: Profile JIT Compilation and Inline Decisions Using PrintCompilation Flags

In Day 26, we implemented a Token-Bucket backpressure mechanism to prevent our AstraKV socket buffers from overflowing under load. By throttling incoming requests, we stopped the server from crashing due to `OutOfMemoryError` or kernel-level packet drops. However, even with backpressure, your server’s throughput might still hit a “performance ceiling” where CPU usage is high, but request latency spikes unpredictably.

Today, we look inside the JVM’s engine room: the Just-In-Time (JIT) compiler. We will learn how to observe the JVM’s decision-making process as it transforms your bytecode into machine code, specifically focusing on **inlining**—the most critical optimization for high-performance storage engines.

## The Problem: The “Black Box” performance Plateau

In a system like AstraKV, your `StorageEngine` and `NetworkHandler` methods are called millions of times per second. If the JIT compiler fails to inline these hot methods, the overhead of repeated method calls (stack frame allocation, register saving) becomes a significant tax on your throughput.

Consider the “LMAX Disruptor” architecture. At the scale of hundreds of millions of requests, the difference between an inlined method call and a virtual method call is the difference between meeting your latency SLA or timing out. When the JIT compiler makes a “bad” decision—or when it is constantly re-compiling due to class loading—your system experiences “jitter,” or intermittent latency spikes that are nearly impossible to debug without visibility into the compilation log.

## The Mechanism: JIT Compilation and Inlining

The HotSpot JVM uses two compilers: C1 (Client, fast, basic optimization) and C2 (Server, slow, aggressive optimization). When a method is “hot,” C2 takes over. The most powerful optimization C2 performs is **inlining**: replacing a method call with the actual body of the method.

“`java
// Conceptual: The JIT compiler transforms this…
public void handle(Request req) {
if (this.validate(req)) {
this.process(req);
}
}

// …into this (machine code level):
public void handle(Request req) {
// Inlined body of validate()
if (req.isValid()) {
// Inlined body of process()
this.storage.put(req.key, req.val);
}
}
“`

By inlining, we eliminate the jump instruction and allow the CPU to perform better branch prediction and instruction scheduling. If your critical path methods are too large or contain too many polymorphic call sites, the JIT compiler will refuse to inline them, leaving your system running at “interpreted” or “partially optimized” speed.

## The Production Stakes: The “Warm-up” Outage

In 2013, engineers at a major financial firm observed that their trading engines would periodically stall for 500ms every time they deployed a new configuration. It turned out that the configuration change triggered the loading of a new class, which forced the JIT compiler to de-optimize existing code and re-compile the hot path. This is the “warm-up” problem: systems are not production-ready the moment they start; they are only ready once the JIT has finished its work. Understanding `PrintCompilation` allows you to see exactly when your system is “warmed up.”

## Observing the Compiler

We will use the `-XX:+PrintCompilation` and `-XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining` flags. These flags turn the JVM’s internal decision-making process into a readable log.

“`java
// Example of what we are looking for in the logs:
// 123 45 n com.astrakv.StorageEngine::put (15 bytes)
// 124 46 % com.astrakv.NetworkHandler::loop @ 10 (50 bytes)
“`

The output tells you:
1.**Timestamp/ID:** When the compilation happened.
2.**Compiler:** `n` (native), `s` (synchronized), or `%` (OSR – On-Stack Replacement).
3.**Method:** The signature being compiled.

## Failure Demo: The “De-optimization” Trap

Today, we will deliberately break the JIT’s ability to optimize by introducing a “polymorphic” call site that changes its implementation type frequently. You will see the JIT compiler work hard to inline a method, only to be forced to “de-optimize” it when it realizes the implementation is no longer stable. This causes a massive, observable drop in throughput.

## Production Reality vs. Laptop Constraints

In production, you would never keep `PrintCompilation` enabled, as it creates significant I/O overhead. You use it during load testing to verify that your “hot” methods (like `put` and `get` in AstraKV) are being inlined by C2. On your laptop, we will use it to learn the signal. We ignore the complexity of tiered compilation levels (C1 vs C2) for now, focusing only on the *fact* of compilation.

### Assignment

Your task is to identify the “hot” method in your `StorageEngine` implementation.
1.Run the server with `-XX:+PrintCompilation`.
2.Use a simple loop to call `put()` 1,000,000 times.
3.Capture the output to a file and grep for your class name.
4.**Success Criteria:** You must find the line where your `put` method is compiled by the C2 compiler (indicated by the `4` in the compilation ID column).

### Solution Hints
*The JVM needs a “warm-up” period. A single call won’t trigger C2. You need a loop of at least 10,000–50,000 iterations for the JIT to consider a method “hot.”
*If you don’t see your method, it might be too small (the JVM inlines it automatically without a separate compilation log entry) or too large (exceeding the default inline budget).
*Check the `PrintInlining` output to see if the JVM rejected your method for being “too big.”

Hands-On System Design Tutorial

# Day 26: Implement Token-Bucket Backpressure

In Day 25, we successfully exposed our AstraKV storage engine over a TCP socket using Java NIO. You built a server that can accept client connections and process `GET` and `PUT` commands. However, that server has a fatal flaw: it is “too eager.” It will accept every byte a client sends as fast as the kernel buffer allows, potentially exhausting memory or CPU cycles before the storage engine can even acknowledge the request. Today, we fix this by implementing a Token-Bucket algorithm to enforce backpressure.

### The Problem: The “Firehose” Effect
When a client sends data faster than your server can process it, the OS kernel queues those bytes in the TCP receive buffer. If you don’t read them, the buffer fills up, and the TCP window size drops to zero, forcing the client to pause. This is a form of passive backpressure. But what happens if you *are* reading, but your internal logic (disk I/O or index lookups) is slower than the network? You end up with an unbounded queue of pending requests in your application heap.

This is exactly what led to the famous “Cascading Failure” scenarios in systems like those documented by AWS during their early DynamoDB outages: when one node slows down, the upstream clients retry more aggressively, flooding the struggling node until it crashes, causing its neighbors to take the load and crash in turn.

### The Mechanism: Token Bucket
To prevent this, we introduce a **Token Bucket**. Imagine a bucket that holds a fixed number of “tokens.” Each request consumes one token. Tokens are added to the bucket at a constant rate. If the bucket is empty, the server refuses the request (or forces the client to wait).

***Bucket Capacity ($B$):** Allows for short bursts of traffic.
***Refill Rate ($R$):** The long-term sustainable throughput of your system.

If your storage engine can handle 1,000 requests per second, you set $R=1000$. If you get a sudden spike, the $B$ tokens allow you to absorb it, provided the average stays within $R$.

### Architectural Integration
We place the Token Bucket directly in the `RequestProcessor` pipeline, just after the `SocketChannel` reads the bytes but before the `StorageEngine` executes the command.

[DIAGRAM: Client -> Socket Buffer -> [Token Bucket] -> Storage Engine]
*Annotation: If Bucket == 0, return 429 Too Many Requests or close the connection to signal backpressure.*

### Conceptual Implementation
You don’t need a complex library for this. A simple `AtomicLong` and a timestamp check suffice for a single-threaded reactor:

“`java
public class TokenBucket {
private final long capacity;
private final long refillRatePerNanos;
private long tokens;
private long lastRefillTimestamp;

public synchronized boolean tryConsume() {
refill();
if (tokens > 0) {
tokens–;
return true;
}
return false;
}

private void refill() {
long now = System.nanoTime();
long delta = now – lastRefillTimestamp;
tokens = Math.min(capacity, tokens + (delta / refillRatePerNanos));
lastRefillTimestamp = now;
}
}
“`

### Trade-offs: Why not a Semaphore?
You might be tempted to use a `java.util.concurrent.Semaphore`. A semaphore is great for limiting *concurrency* (how many requests are running *right now*). A Token Bucket limits *rate* (how many requests per second). In storage engines, we care about rate because disk I/O latency is throughput-dependent. If you use a semaphore, you might allow a burst of 100 requests that all hit the disk simultaneously, causing IOPS saturation and latency spikes.

### The Failure Demo
Today, you will use a client script to blast 5,000 requests per second at a server configured to handle only 500. Without the bucket, your JVM heap will grow until an `OutOfMemoryError` or the latency will climb into the seconds as the GC struggles to clean up the queue of pending objects. With the bucket, you will observe the server rejecting requests, keeping the system stable at the defined 500 RPS.

### Looking Ahead
Tomorrow, in Day 27, we will look at how the JVM JIT compiler decides to inline our `tryConsume()` method. Because we are now doing math in the hot path of every request, we need to ensure our Token Bucket doesn’t become a performance bottleneck itself.

—

### Assignment
1.**Integrate:** Add the `TokenBucket` class to your `astra.net` package.
2.**Instrument:** Modify your `NioServer` to instantiate a bucket with a capacity of 100 and a rate of 500 RPS.
3.**Reject:** If `tryConsume()` returns false, send a custom “429 Too Many Requests” response back to the client.
4.**Verify:** Run the load test provided in the repo. Ensure that while the client reports failures, the server process remains responsive and memory usage stays flat.

### Solution Hints
*Don’t use `Thread.sleep` to refill; use the delta-time approach shown in the snippet.
*Your `NioServer` loop should check the bucket before calling `storageEngine.execute()`.
*If you see “Connection Reset” errors, your client might be closing too fast; ensure you handle the server’s response even when it’s a 429.

Hands-On System Design Tutorial

# Day 25: Expose the Storage Engine Over a TCP Socket Server Using Java NIO

Yesterday, we built a memory-bounded storage engine that handles data safely by pinning buffers and managing its own memory outside the unpredictable influence of the JVM Garbage Collector. Today, we bridge the gap between that engine and the outside world by building a TCP server using Java NIO (New I/O).

### The Concept: Non-Blocking Event Loops vs. Thread-per-Connection
In traditional Java networking (the old `java.io` package), every incoming TCP connection requires a dedicated thread. If you have 10,000 clients, you have 10,000 threads. This is a disaster: the kernel spends more time context-switching between threads than executing your code, and the memory overhead of 10,000 thread stacks—even at 1MB each—will crash your JVM long before you hit your performance goals.

We use **Java NIO** to solve this. Instead of one thread per client, we use a single “Selector” thread that monitors thousands of sockets for events (like “data ready to read”). This is the same architectural pattern used by high-performance systems like Netty, Kafka, and the internal RPC layers of distributed databases like Apache Cassandra.

### The Production Stakes: The “Thundering Herd” and Socket Saturation
When you expose a storage engine directly to a network, you lose control over the arrival rate of requests. If your engine takes 5ms to process a request and 1,000 clients connect simultaneously, your queue will grow unbounded, leading to **latency inflation**.

A famous example of this is the “TCP Backlog” failure documented in various Linux kernel tuning guides: when the server’s listen queue fills up, the kernel begins dropping new connections (SYN drops). If your application logic doesn’t handle the transition between the network buffer and the storage engine correctly, you can end up with partial writes or, worse, memory exhaustion as incoming requests buffer indefinitely in the JVM heap.

### Architecture: The Reactor Pattern
We implement a simplified **Reactor Pattern**.

1.**The Acceptor:** Monitors the `ServerSocketChannel` for new connection attempts.
2.**The Reactor:** A single thread running a `Selector` loop. It polls the OS for channels that have data ready to be read.
3.**The Processor:** Once data is present, we read it into the buffer pool we built in Day 24, route the command to our storage engine, and write the response back.

### The Failure Demo: The “Slow Consumer”
In the Implementation Guide, you will trigger a failure by intentionally pausing the processing logic while flooding the server with requests. You will observe the `Selector` continue to accept connections, but the “Read” operations will start piling up in the kernel’s receive buffer. Eventually, the client-side TCP window will close, and the client will hang—this is the network performing **natural backpressure**. Tomorrow, we will learn how to manage this explicitly before the kernel has to force it on us.

### Trade-off: Why not use Netty?
You might ask: “Why build this raw NIO instead of using Netty?” Netty is an incredible library, but it hides the `Selector` and `ByteBuffer` management. By building this today, you learn how to debug `Selector` starvation and buffer leaks. In production, you would use Netty to save development time, but you would use the knowledge from today to tune its `EventLoopGroup` threads and `ByteBufAllocator`.

### Assignment: The Throughput Test
Your task is to modify the `AstraKVServer` to log the number of bytes processed per second.
1.Use the provided `load-tester` script to send 10,000 small GET requests.
2.Observe the latency.
3.**The Challenge:** Introduce a `Thread.sleep(10)` inside the `processCommand` method. Run the test again. Observe how the server stops responding to new requests once the internal queue depth is exceeded.

### Solution Hints
-To log throughput, use a `LongAdder` to count bytes read in the `handleRead` method.
-To see the “hang,” use `netstat -an | grep 8080` to see the `Recv-Q` (Receive Queue) filling up. If `Recv-Q` stays at a high number, your server is not reading fast enough.

Hands-On System Design Tutorial

# Day 24: Building a Memory-Bounded Storage Engine with Automatic GC-Friendly Buffer Pools

In our last session, we triggered an `OutOfMemoryError` by naively buffering incoming write requests into the JVM heap. We saw how quickly the Garbage Collector (GC) becomes the bottleneck when it has to manage millions of short-lived objects. Today, we move beyond the heap. We are going to build a **Buffer Pool**—a fixed-size, off-heap memory region—to act as the gatekeeper for AstraKV’s storage engine.

### The Problem: When GC Becomes Your Worst Enemy
In high-throughput systems like Apache Kafka or LinkedIn’s Espresso, if you allocate every incoming request as a new Java object on the heap, you aren’t just storing data; you are creating work for the GC. At 100,000 requests per second, the GC will spend more time marking and sweeping your objects than your application will spend writing them to disk. This is the “Stop-the-World” tax.

By moving our buffers off-heap using `java.nio.ByteBuffer.allocateDirect()`, we move the memory out of the reach of the GC’s reachability analysis. The tradeoff? We lose automatic memory management. If we don’t recycle these buffers, we leak memory at the OS level, eventually crashing the process with a native `OutOfMemoryError` that no heap dump can explain.

### The Architecture: The Buffer Pool
A Buffer Pool is a pre-allocated slab of memory divided into fixed-size “slots.” Instead of `new byte[]`, our engine checks out a slice of this slab. When the storage operation completes, the slice is returned (checked in) to the pool.

#### Component Interaction
– **The Pool:** A `ConcurrentLinkedQueue` or `ArrayBlockingQueue` holding available `DirectByteBuffer` instances.
– **The Engine:** Requests a buffer from the pool before parsing the network packet.
– **The Lifecycle:** Once the data is flushed to the WAL (Write Ahead Log), the engine returns the buffer to the pool.

“`java
// A simple pool entry retrieval
public ByteBuffer acquire() {
ByteBuffer buffer = pool.poll();
if (buffer == null) {
// We are at capacity, apply backpressure or block
throw new BufferPoolExhaustedException();
}
buffer.clear();
return buffer;
}
“`

### The Trade-off: Fixed Capacity vs. Dynamic scalability
We are choosing a **Fixed-Size Buffer Pool**.
***Alternative:** Dynamic allocation. It’s easier to implement but leads to fragmentation and non-deterministic latency.
***Why we choose Fixed-Size:** It provides an inherent **backpressure mechanism**. If the pool is empty, the system cannot accept more work. This prevents the “death spiral” where an overloaded system crashes itself by trying to buffer everything at once.

### Failure Demo: The “Memory Leak” Simulation
In the implementation guide, you will run a load test that deliberately leaks buffers by failing to “check them in.” You will observe the system reach its hard limit and reject new requests. This is not a failure; this is the system protecting itself from total collapse.

### Scaling to Production
At scale, you don’t just have one pool. You might have pools for different sizes (e.g., 4KB, 16KB, 64KB) to prevent internal fragmentation. Remember: off-heap memory is invisible to standard JVM monitoring tools like `jstat`. You must monitor `process_resident_memory_bytes` at the OS level to ensure your pool isn’t leaking.

As we move toward Day 19, we will use this pool to measure how much “garbage” we have removed from the heap, allowing us to tune the JVM’s GC parameters for a much smaller, more stable heap footprint.

—

### Assignment
1.**Instrument the Pool:** Add a counter to the `acquire` and `release` methods to track the current “in-use” count.
2.**Break the Pool:** Create a test case where you acquire 100 buffers but only release 90. Observe the log output as the system hits the limit.
3.**Verify:** Use the provided `verify_pool.sh` to confirm that under a sustained load of 5,000 requests, the heap usage remains flat while the native memory usage stays within your configured bounds.

### Solution Hints
*Use a `Semaphore` initialized to the pool size to implement the blocking logic when the pool is empty.
*Ensure that every `ByteBuffer` is `clear()`ed before reuse to avoid cross-pollination of data from previous requests.
*Check the repository’s `BufferPool.java` for the base implementation of the `acquire`/`release` cycle.

Hands-On System Design Tutorial

# Day 23: Trigger an OutOfMemoryError by Saturating the Heap and Analyze the Heap Dump

In our previous session, we used GC logs to compare G1GC and ZGC, observing how different collectors handle high-allocation pressure. Today, we confront the “end-state” of heap management: the `OutOfMemoryError` (OOME).

When a Java application hits its heap limit, it doesn’t just crash gracefully. It often enters a “death spiral” where the GC works at 100% CPU capacity trying to reclaim a few bytes, causing latency to spike to seconds before the JVM finally throws an OOME. Understanding this transition is critical for building robust storage engines.

### The Stakes: The “Death Spiral”
In production, an OOME is rarely a clean exit. Consider the 2012 GitHub outage, where a memory leak in a Ruby-based component caused the system to thrash in GC before failing. In Java, this often manifests as “Stop-the-World” (STW) pauses that grow exponentially as the heap fills. If your storage engine doesn’t have an explicit memory-bounding strategy, it will eventually consume the entire container limit, triggering the OOM Killer at the OS level, which leaves behind no heap dump and zero diagnostic context.

### The Concept: Heap Saturation and Forensic Capture
Today, we will intentionally force a heap exhaustion event. The goal is to observe the transition from “healthy allocation” to “GC thrashing” to “OOME,” and then extract a heap dump to diagnose the culprit.

We rely on the JVM’s ability to generate a snapshot of the heap at the moment of death. This is the difference between guessing why a system failed and knowing exactly which object type consumed your memory.

### Anatomy of a Memory Leak
In AstraKV, we represent data as `byte[]` arrays. If we store these in a `HashMap` without a bound or an eviction policy, we create a classic memory leak.

“`java
// A simple, dangerous cache that never evicts
public class UnboundedCache {
private final Map storage = new HashMap<>();

public void put(String key, byte[] value) {
// No capacity check: memory will grow until OOME
storage.put(key, value);
}
}
“`

The logic above is simple, but it ignores the fundamental law of distributed systems: **Finite resources require explicit bounds.** In production, an unbounded cache is a time bomb.

### The Failure Demo: The “OOM Trap”
We will run a workload that inserts data into an unbounded map while monitoring the process. You will observe the JVM attempting to reclaim space, the GC frequency increasing, and finally, the process terminating with a heap dump file.

### Production Realities
At hyperscale, we rarely rely on default JVM behavior for memory management. We use:
1.**Off-heap memory:** Using `java.nio.ByteBuffer.allocateDirect()` to move data outside the GC-managed heap, preventing GC pauses for large caches.
2.**Backpressure:** Preventing the system from accepting new requests when memory utilization exceeds a threshold (e.g., 80%).
3.**Soft/Weak References:** Allowing the GC to reclaim cache entries under extreme pressure, though this is often too unpredictable for high-performance storage engines.

We choose to learn about heap saturation today because before we can build a memory-bounded engine tomorrow, we must be able to recognize the exact moment a system becomes “unbounded.”

—

### Assignment
1.**Trigger:** Run the `OOMGenerator` utility in the provided repository. This will fill an `ArrayList` with large objects until the heap is exhausted.
2.**Observe:** Watch the console output to see the GC activity log messages transition from short pauses to multi-second “Full GC” events.
3.**Analyze:** Use `jhat` or VisualVM to open the resulting `.hprof` file. Identify the class consuming the most memory.
4.**Compare:** Run the same test with `-Xmx128m` versus `-Xmx512m` and observe how the time-to-failure correlates with heap size.

### Solution Hints
– The heap dump will be generated in the project root if you set the JVM flag `-XX:+HeapDumpOnOutOfMemoryError`.
– If the JVM doesn’t crash fast enough, increase the `byte[]` size in the `OOMGenerator` to consume memory faster.
– Use `jvisualvm` (or `jhat`) to inspect the “Heap Histogram” to see which object type (likely `byte[]`) dominates the heap.

Hands-On System Design Tutorial

# Day 22: Profile Garbage Collection Pauses Using GC Logs and Compare G1GC with ZGC

In our previous session, we used JMH to quantify the performance gap between off-heap memory and the standard JVM heap. We saw that while heap storage is easier to manage, it introduces a hidden tax: the Garbage Collector (GC). Today, we move from micro-benchmarking to macro-observability. We will instrument AstraKV to visualize exactly when and why the GC stops our world, and compare the “stop-the-world” behavior of the traditional G1GC against the low-latency ZGC.

### The Problem: The “Death Spiral” of Latency
In distributed storage systems, tail latency is often more important than average throughput. If 99.9% of your requests take 1ms, but 0.1% take 500ms due to a GC pause, your downstream clients will eventually time out, retry, and trigger a cascading failure.

A famous example of this occurred in the early days of LinkedIn’s transition to large heap JVMs. They observed “GC pauses” that grew linearly with heap size, eventually causing nodes to drop out of the cluster because the heartbeat thread couldn’t get CPU time. They had to abandon large heaps in favor of many smaller JVM instances—a complexity tax paid solely because the GC couldn’t keep up with the object churn.

### The Mechanism: GC Logs as a System Heartbeat
The JVM provides internal logging to track every reclamation event. By enabling Unified Logging (`-Xlog:gc*`), we move from guessing why a system stalled to having a timestamped audit trail of heap occupancy, promotion rates, and pause durations.

The core concept here is **Pause Time vs. Throughput**. G1GC (Garbage First) is a regionalized collector that prioritizes throughput, often resulting in pauses that correlate with the number of live objects. ZGC (Z Garbage Collector), by contrast, performs almost all its work concurrently with the application threads, keeping pause times sub-millisecond regardless of heap size.

### Architecture: Observing the Lifecycle
In AstraKV, we will configure the JVM to emit structured logs that track the “Young Gen” and “Old Gen” transitions.

“`java
// Conceptual snippet: How AstraKV interacts with the GC
// We don’t trigger GC manually; we monitor it via JMX
public void reportGCMetrics() {
List beans = ManagementFactory.getGarbageCollectorMXBeans();
for (GarbageCollectorMXBean bean : beans) {
// This captures the “Collection Time” metric we’ll use in our dashboard
long totalTime = bean.getCollectionTime();
long count = bean.getCollectionCount();
System.out.printf(“GC: %s, Count: %d, Total Time: %dms%n”, bean.getName(), count, totalTime);
}
}
“`

When you look at these logs, you are looking at the “breath” of your system. A healthy system has consistent, short pauses. A system in a “GC Death Spiral” shows increasing pause frequency and duration as the heap becomes fragmented or exhausted.

### The Failure Demo: The “Allocation Wall”
We will deliberately saturate AstraKV with high-frequency, short-lived object allocations to force the GC into a frenzy. You will observe that G1GC pauses grow noticeably as the heap fills, whereas ZGC maintains a flat latency profile, even as the system struggles to keep up with the allocation rate.

### Trade-offs: Why not use ZGC everywhere?
If ZGC is so fast, why does G1GC still exist? ZGC requires more CPU overhead to maintain its “colored pointers” and concurrent marking. In a throughput-heavy batch processing job, G1GC might finish 10% faster overall, even if it has a few 50ms pauses. In a real-time storage engine like AstraKV, we prioritize the sub-millisecond pause.

### Looking Ahead
Today we observe the pauses. Tomorrow, in Day 23, we will intentionally push the heap beyond its limit to trigger an `OutOfMemoryError`, capture a heap dump, and use tools like Eclipse MAT to identify which specific AstraKV objects are pinning our memory and preventing reclamation.

—

### Assignment
1.**Instrument:** Modify your run-script to include `-Xlog:gc:file=gc.log`.
2.**Observe:** Run the `AstraKVLoadGenerator` (provided in the repo) under both G1GC and ZGC.
3.**Compare:** Use `grep` to find the “Pause” duration lines in your `gc.log` files.
4.**Analyze:** Identify the maximum pause time for each collector.

### Solution Hints
*To switch collectors, use `-XX:+UseG1GC` or `-XX:+UseZGC`.
*Look for lines in the GC log containing `Pause Young` or `Pause Remark`.
*The `AstraKVLoadGenerator` uses a memory-heavy workload designed to fill the Eden space rapidly. If you don’t see pauses, increase the `-Xmx` (heap size) to be smaller to force more frequent collections.

Hands-On System Design Tutorial

# Day 21: Benchmarking the Boundary between Heap and Off-Heap

In Day 20, you refactored AstraKV to store its payload segments in `DirectByteBuffer` instances, effectively moving your primary data storage out of the JVM’s garbage-collected heap. By doing this, you successfully reduced the “stop-the-world” pause times caused by the GC scanning millions of objects. Today, we confront the reality of that decision: moving data off-heap is not a free lunch. We are trading GC pressure for explicit memory management and potential cache-miss overhead.

To make informed engineering decisions, you cannot rely on intuition or simple `System.currentTimeMillis()` measurements. You need the Java Microbenchmark Harness (JMH).

### The Production Stakes: The “Hidden” Latency Tax
In high-throughput systems like Apache Kafka or Cassandra, developers often move data off-heap to manage massive caches (dozens of gigabytes) without triggering long GC cycles. However, there is a catch: if you mismanage your memory lifecycle, you can cause native memory fragmentation or introduce significant latency spikes during allocation. In 2014, the LinkedIn engineering team documented how improper memory management in their storage layers led to “silent” latency degradation—where the system didn’t crash, but P99 latencies climbed because the CPU spent more time managing memory pointers than serving requests. Today, you will build the tool to detect if your off-heap implementation is actually faster, or if you’ve just traded one performance tax for another.

### The Concept: Measuring “Hot” Code Paths
JMH is designed to prevent the two most common sins of microbenchmarking:
1.**Dead Code Elimination:** If the JVM detects that your benchmark’s result isn’t used, it will optimize the entire method away.
2.**Constant Folding:** If the JVM detects your input is constant, it will pre-calculate the result at compile time.

JMH forces the JVM to treat your code as a real, live workload by using `Blackhole` objects to consume results and `@State` objects to simulate concurrent access.

### Architecture: The Benchmark Harness
Our benchmark compares three storage strategies for AstraKV:
1.**Heap-based:** A standard `byte[]` array.
2.**Off-Heap (Direct):** A `DirectByteBuffer`.
3.**Off-Heap (Unsafe):** Raw pointer access via `sun.misc.Unsafe`.

### Implementation Snippet: The Benchmark State
The following snippet shows how we define the state for our benchmark. Note the use of `@Setup` to initialize the memory before the measurement begins, ensuring we aren’t benchmarking the cost of memory allocation itself.

“`java
@State(Scope.Thread)
public class StorageBenchmark {
private ByteBuffer directBuffer;
private byte[] heapArray;

@Setup
public void setup() {
directBuffer = ByteBuffer.allocateDirect(1024);
heapArray = new byte[1024];
}

@Benchmark
public byte benchmarkHeap() {
return heapArray[512];
}

@Benchmark
public byte benchmarkDirect(Blackhole bh) {
bh.consume(directBuffer.get(512));
return 0;
}
}
“`

### Failure Demo: The “Memory Leak” Simulation
In the Implementation Guide, you will trigger a scenario where you allocate native memory within the benchmark loop without calling `cleaner.clean()`. You will observe the resident set size (RSS) of your process climbing until the OS invokes the OOM killer. This demonstrates why off-heap memory is a “manual” responsibility: the JVM doesn’t know it exists, so it cannot reclaim it.

### Forward Link
After today, you will have a clear performance profile of your storage engine. In **Day 22: Profile Garbage Collection Pauses Using GC Logs and Compare G1GC with ZGC**, we will use this knowledge to tune the JVM’s GC collector settings to handle the remaining heap objects while your off-heap data sits safely outside the collector’s reach.

Hands-On System Design Tutorial

# Day 20: Refactor Storage Payloads to Off-Heap ByteBuffers

In Day 19, we used `jmap` and `jhat` to visualize how your AstraKV write-load was bloating the JVM heap. You saw firsthand how millions of small `byte[]` objects created a “GC tax”—the garbage collector spent more time tracing your objects than your application spent processing requests. Today, we break that cycle by moving our storage payloads out of the managed heap entirely.

### The Problem: The GC Tax
When you store data on the Java heap, the Garbage Collector (GC) must track every single object. If you have 10 million keys in AstraKV, that’s 10 million objects the GC must scan during every “mark” phase. Even if the data is static, the GC overhead scales linearly with the number of objects, not just the volume of data.

In production systems like **Apache Cassandra** or **Kafka**, storing massive volumes of data on the heap leads to “Stop-the-World” pauses that can last seconds. When the GC pauses, your network buffers fill up, TCP connections time out, and your p99 latency spikes.

### The Mechanism: Off-Heap Memory
We will use `java.nio.ByteBuffer.allocateDirect()`. This allocates memory in the process’s native memory space, outside the JVM’s managed heap.

The JVM GC does not scan this memory. It treats it as an opaque blob. This is a double-edged sword: you get consistent latency regardless of data volume, but you lose automatic memory management. If you allocate off-heap memory and lose the reference, you have a native memory leak that `jmap` won’t show you. You must manually manage the lifecycle.

### The Architecture: Off-Heap Slab
Instead of creating a new object per write, we will implement a `SlabAllocator`. Think of this like a physical warehouse: you have a large pre-allocated floor space (the `DirectByteBuffer`), and you manage the “slots” yourself.

“`java
// Conceptual snippet: Allocating a slab
public class OffHeapStorage {
private final ByteBuffer buffer;

public OffHeapStorage(int capacityBytes) {
// This memory lives in the OS process space, not the JVM Heap
this.buffer = ByteBuffer.allocateDirect(capacityBytes);
}

public void write(int offset, byte[] data) {
buffer.put(offset, data);
}
}
“`

### Why This Matters
By moving payloads off-heap, your GC frequency should drop significantly. You are trading complexity (manual index management) for predictability.

**Trade-off:** If you require ultra-low latency but have highly variable data sizes, off-heap management becomes complex due to fragmentation. In those cases, a “pooled” approach (reusing fixed-size buffers) is often preferred over a contiguous slab.

### The Failure Demo: The Silent Leak
In the Implementation Guide, you will run a load test that fills memory. We will then “break” the system by intentionally failing to recycle a buffer. You will observe the process’s Resident Set Size (RSS) grow in `top` while the JVM heap usage remains flat. This is the hallmark of a native memory leak—the JVM thinks it’s healthy, but the OS is about to trigger the OOM Killer.

### Production Stakes
This approach mirrors how **Netty** handles network buffers. In high-throughput systems, copying data from the kernel to the JVM heap and back is a performance killer. By using direct buffers, you enable “Zero-Copy” patterns, where the OS can move data directly from the network card to your application memory.

—

### Assignment
1.Modify your existing `StorageEngine` to use a `DirectByteBuffer` instead of a `HashMap`.
2.Implement a simple “offset-based” index that maps your keys to a `long` position in the `ByteBuffer`.
3.Verify the reduction in GC activity using `jstat -gc 1000`.

Hands-On System Design Tutorial

# Day 19: Measuring Heap vs. Stack Allocations Under High Write Load

In our previous lesson, we successfully integrated a Concurrent Write-Ahead Log (WAL) with an in-memory index for AstraKV. We now have a system that promises durability by flushing writes to disk before acknowledging them. However, as we scale, you will notice that the JVM’s garbage collector (GC) begins to fight your application for CPU cycles. Today, we focus on the “why” behind these pauses.

### The Problem: The Hidden Cost of Object Allocation
In Java, we are taught to treat objects as cheap abstractions. In systems engineering, we treat them as memory-pressure events. Every time you create an `Entry` object or a `LogRecord` for your WAL, you are allocating memory on the heap. When that memory is no longer reachable, the GC must traverse the object graph to reclaim it.

At high request rates, this creates “allocation churn.” If your allocation rate exceeds the GC’s ability to reclaim, you hit a “Stop-the-World” (STW) event. This is the primary reason for tail-latency spikes in systems like Apache Cassandra or Kafka. When the GC pauses, your network buffers fill up, your WAL threads block, and your P99 latency goes off a cliff.

### The Concept: Escape Analysis and Allocation Profiling
The JVM tries to be smart. Through **Escape Analysis**, the Just-In-Time (JIT) compiler determines if an object is only used within a single method. If it is, the JVM can perform “Scalar Replacement,” allocating the object’s fields directly on the stack (or even in registers), bypassing the heap entirely.

If an object “escapes” (is returned, stored in a static field, or passed to a long-lived concurrent thread), it must go to the heap. Your goal as a systems engineer is to maximize stack allocation and minimize heap transit.

### The Anchor: The “Stop-the-World” Reality
Consider the famous 2012 LinkedIn post-mortem on their transition from Java to C++ for certain components. They hit GC walls where 30-second pauses were common. They weren’t just “bad coders”; they were victims of high-volume object churn that the JVM at the time couldn’t optimize away. While modern GCs (ZGC, Shenandoah) are better, they are not magic. If you allocate 1GB/sec, you will eventually pay the tax.

### Architecture: Monitoring the Churn
We will use `jstat` and `async-profiler` to visualize this. We are not just looking at “how much memory is used,” but “how much memory is allocated per second.”

“`java
// Conceptual snippet: Tracking allocation in the Write Path
public void write(byte[] key, byte[] value) {
// Each Entry instantiation here is a potential heap allocation
Entry entry = new Entry(key, value);
wal.append(entry);
index.put(key, entry.offset());
}
“`

If `Entry` escapes the `write` method (which it does, as it’s referenced by the index), the JVM *must* put it on the heap. Understanding this boundary is the difference between a system that handles 10k ops/sec and one that handles 100k.

### The Failure Demo: The GC Death Spiral
In the Implementation Guide, you will run a load test that saturates the system. We will then trigger a configuration change that forces the JVM to perform frequent young-generation GCs. You will observe the “sawtooth” pattern of the heap usage and, eventually, a sustained GC pause that causes your client-side latency to jump from 1ms to 500ms.

### Trade-offs: Heap vs. Off-Heap
We choose to use the Heap for now because it is safe and idiomatic. The alternative is manual memory management using `DirectByteBuffers` (Off-Heap). Off-heap is faster and invisible to the GC, but it introduces the risk of memory leaks and segmentation faults—the very things Java was designed to prevent. We will move to Off-Heap in Day 20, but only after we have mastered the art of measuring exactly what the heap is doing.

### Assignment: The Allocation Audit
1.**Run the Load Generator:** Use the provided script to push 50,000 writes/sec into AstraKV.
2.**Observe:** Use `jstat -gc` to watch the `YGC` (Young Generation GC) count increment.
3.**Analyze:** Identify the point where the `Eden` space fills up and triggers a collection.
4.**Break:** Increase the `Entry` size by 10x and observe how the `YGC` frequency increases, directly correlating to the throughput drop.

### Solution Hint
Look at the `Eden` space utilization in `jstat`. If your `Eden` space is 256MB and you are allocating 1GB/sec, you should expect a `YGC` every 250ms. If you see them happening faster, your system is “allocating” more than just your data—it’s allocating temporary objects you didn’t account for.

Hands-On System Design Tutorial

# Day 18: Integrate the Concurrent Write-Ahead Log with the In-Memory Index

In Day 17, we diagnosed thread allocation deadlocks and extracted thread dumps to see where threads got stuck under heavy, uncoordinated resource contention. Today, we transition from diagnosing thread failures to building a highly concurrent, crash-consistent storage engine.

We will integrate two core components of AstraKV: the disk-backed **Write-Ahead Log (WAL)** and the volatile **In-Memory Index (MemTable)**. This integration is the foundation of any crash-resilient database. In Day 13, we will spin up multiple platform threads to hammer this integrated store concurrently, exposing subtle race conditions if our index-WAL boundary is not properly synchronized. Today, we focus on establishing the atomic write path and crash-recovery mechanics that guarantee durability without killing concurrent write throughput.

## The Production Stakes: The Cost of Out-of-Order Writes

When a database client executes a write, the engine must update both the persistent log (WAL) on disk and the fast, sorted index (MemTable) in memory.

In 2015, early deployments of distributed databases frequently suffered from “phantom updates” or silent data corruption during sudden power losses. If a system updates the in-memory MemTable *before* the write is fully flushed and fsynced to the WAL, concurrent readers can immediately see the new value. However, if the server crashes a millisecond later, that un-flushed write vanishes from disk. Upon reboot, the database recovers to an older state. The client, which received a success acknowledgment, discovers its data has vanished—a direct violation of ACID durability.

## Deep Dive: The Coordination Engine

Let us examine the core coordinate write path. The class `AstraKVStore` manages this orchestration. We must write to the `FileChannel` sequentially to prevent log corruption, but we want to allow concurrent threads to update the index as soon as their respective disk writes complete.

“`java
public void put(String key, String value) throws IOException {
byte[] keyBytes = key.getBytes(StandardCharsets.UTF_8);
byte[] valBytes = value.getBytes(StandardCharsets.UTF_8);

// Step 1: Append to the physical log under a fine-grained write lock
long lsn;
writeLock.lock();
try {
lsn = wal.append(keyBytes, valBytes);
} finally {
writeLock.unlock();
}

// Step 2: Update the in-memory index concurrently without holding the write lock
memTable.put(key, value);
}
“`

This design ensures that disk serialization is the only bottleneck. Multiple threads can concurrently traverse and update the lock-free skip-list of the MemTable while another thread is waiting for the OS to complete its disk write.

### Recovery Mechanics: Scanning the Log

Upon startup, the store must transition from an empty state to a fully restored state by parsing the binary WAL. We must handle corrupt or partially written entries at the end of the log (which occur when a crash happens mid-write).

“`java
public void recover() throws IOException {
wal.seekToStart();
while (wal.hasNext()) {
try {
LogEntry entry = wal.readNext();
if (entry.isValid()) {
memTable.put(entry.key(), entry.value());
} else {
// Stop recovery at the first corrupted record (typically a partial write at crash time)
break;
}
} catch (EOFException e) {
break;
}
}
}
“`

## Assignment: Implement Corrupt Record Detection during Recovery

Currently, if the database crashes mid-write, the WAL might contain trailing garbage bytes. Your task is to implement a strict CRC32 checksum validation inside the recovery path.

### Steps to Complete
1.Open `WriteAheadLog.java` and locate the `readNext()` method.
2.Read the recorded CRC32 checksum from the log entry header.
3.Compute the checksum of the read key and value bytes using `java.util.zip.CRC32`.
4.If the computed checksum does not match the recorded checksum, throw a `CorruptedLogException` rather than returning a corrupt entry.
5.In `AstraKVStore.java`, catch the `CorruptedLogException` during recovery, truncate the log file to the last known valid position to discard the partial write, and complete startup.

### Success Criteria
*Running the recovery test on a truncated, corrupted WAL successfully discards the corrupted trailing bytes.
*The store starts up cleanly with only the valid, pre-crash records present in the index.

—

## Solution Hints

To calculate the checksum of the key and value:
“`java
CRC32 crc = new CRC32();
crc.update(keyBytes);
crc.update(valBytes);
long expectedChecksum = crc.getValue();
“`

When writing to the log, ensure you write the checksum as a 64-bit `long` (8 bytes) into the entry header:
“`
[ Magic Byte (1B) ] [ Key Length (4B) ] [ Value Length (4B) ] [ Checksum (8B) ] [ Key Bytes ] [ Value Bytes ]
“`

During recovery, if the checksum fails, truncate the file channel to the position of the start of the corrupt record:
“`java
fileChannel.truncate(lastValidPosition);
“`