Hands-On Lesson

# Day 27: Profile JIT Compilation and Inline Decisions Using PrintCompilation Flags

In Day 26, we implemented a Token-Bucket backpressure mechanism to prevent our AstraKV socket buffers from overflowing under load. By throttling incoming requests, we stopped the server from crashing due to `OutOfMemoryError` or kernel-level packet drops. However, even with backpressure, your server’s throughput might still hit a “performance ceiling” where CPU usage is high, but request latency spikes unpredictably.

Today, we look inside the JVM’s engine room: the Just-In-Time (JIT) compiler. We will learn how to observe the JVM’s decision-making process as it transforms your bytecode into machine code, specifically focusing on **inlining**—the most critical optimization for high-performance storage engines.

## The Problem: The “Black Box” performance Plateau

In a system like AstraKV, your `StorageEngine` and `NetworkHandler` methods are called millions of times per second. If the JIT compiler fails to inline these hot methods, the overhead of repeated method calls (stack frame allocation, register saving) becomes a significant tax on your throughput.

Consider the “LMAX Disruptor” architecture. At the scale of hundreds of millions of requests, the difference between an inlined method call and a virtual method call is the difference between meeting your latency SLA or timing out. When the JIT compiler makes a “bad” decision—or when it is constantly re-compiling due to class loading—your system experiences “jitter,” or intermittent latency spikes that are nearly impossible to debug without visibility into the compilation log.

## The Mechanism: JIT Compilation and Inlining

The HotSpot JVM uses two compilers: C1 (Client, fast, basic optimization) and C2 (Server, slow, aggressive optimization). When a method is “hot,” C2 takes over. The most powerful optimization C2 performs is **inlining**: replacing a method call with the actual body of the method.

“`java
// Conceptual: The JIT compiler transforms this…
public void handle(Request req) {
if (this.validate(req)) {
this.process(req);
}
}

// …into this (machine code level):
public void handle(Request req) {
// Inlined body of validate()
if (req.isValid()) {
// Inlined body of process()
this.storage.put(req.key, req.val);
}
}
“`

By inlining, we eliminate the jump instruction and allow the CPU to perform better branch prediction and instruction scheduling. If your critical path methods are too large or contain too many polymorphic call sites, the JIT compiler will refuse to inline them, leaving your system running at “interpreted” or “partially optimized” speed.

## The Production Stakes: The “Warm-up” Outage

In 2013, engineers at a major financial firm observed that their trading engines would periodically stall for 500ms every time they deployed a new configuration. It turned out that the configuration change triggered the loading of a new class, which forced the JIT compiler to de-optimize existing code and re-compile the hot path. This is the “warm-up” problem: systems are not production-ready the moment they start; they are only ready once the JIT has finished its work. Understanding `PrintCompilation` allows you to see exactly when your system is “warmed up.”

## Observing the Compiler

We will use the `-XX:+PrintCompilation` and `-XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining` flags. These flags turn the JVM’s internal decision-making process into a readable log.

“`java
// Example of what we are looking for in the logs:
// 123 45 n com.astrakv.StorageEngine::put (15 bytes)
// 124 46 % com.astrakv.NetworkHandler::loop @ 10 (50 bytes)
“`

The output tells you:
1.**Timestamp/ID:** When the compilation happened.
2.**Compiler:** `n` (native), `s` (synchronized), or `%` (OSR – On-Stack Replacement).
3.**Method:** The signature being compiled.

## Failure Demo: The “De-optimization” Trap

Today, we will deliberately break the JIT’s ability to optimize by introducing a “polymorphic” call site that changes its implementation type frequently. You will see the JIT compiler work hard to inline a method, only to be forced to “de-optimize” it when it realizes the implementation is no longer stable. This causes a massive, observable drop in throughput.

## Production Reality vs. Laptop Constraints

In production, you would never keep `PrintCompilation` enabled, as it creates significant I/O overhead. You use it during load testing to verify that your “hot” methods (like `put` and `get` in AstraKV) are being inlined by C2. On your laptop, we will use it to learn the signal. We ignore the complexity of tiered compilation levels (C1 vs C2) for now, focusing only on the *fact* of compilation.

### Assignment

Your task is to identify the “hot” method in your `StorageEngine` implementation.
1.Run the server with `-XX:+PrintCompilation`.
2.Use a simple loop to call `put()` 1,000,000 times.
3.Capture the output to a file and grep for your class name.
4.**Success Criteria:** You must find the line where your `put` method is compiled by the C2 compiler (indicated by the `4` in the compilation ID column).

### Solution Hints
*The JVM needs a “warm-up” period. A single call won’t trigger C2. You need a loop of at least 10,000–50,000 iterations for the JIT to consider a method “hot.”
*If you don’t see your method, it might be too small (the JVM inlines it automatically without a separate compilation log entry) or too large (exceeding the default inline budget).
*Check the `PrintInlining` output to see if the JVM rejected your method for being “too big.”

Hands-On Lesson

# Day 26: Implement Token-Bucket Backpressure

In Day 25, we successfully exposed our AstraKV storage engine over a TCP socket using Java NIO. You built a server that can accept client connections and process `GET` and `PUT` commands. However, that server has a fatal flaw: it is “too eager.” It will accept every byte a client sends as fast as the kernel buffer allows, potentially exhausting memory or CPU cycles before the storage engine can even acknowledge the request. Today, we fix this by implementing a Token-Bucket algorithm to enforce backpressure.

### The Problem: The “Firehose” Effect
When a client sends data faster than your server can process it, the OS kernel queues those bytes in the TCP receive buffer. If you don’t read them, the buffer fills up, and the TCP window size drops to zero, forcing the client to pause. This is a form of passive backpressure. But what happens if you *are* reading, but your internal logic (disk I/O or index lookups) is slower than the network? You end up with an unbounded queue of pending requests in your application heap.

This is exactly what led to the famous “Cascading Failure” scenarios in systems like those documented by AWS during their early DynamoDB outages: when one node slows down, the upstream clients retry more aggressively, flooding the struggling node until it crashes, causing its neighbors to take the load and crash in turn.

### The Mechanism: Token Bucket
To prevent this, we introduce a **Token Bucket**. Imagine a bucket that holds a fixed number of “tokens.” Each request consumes one token. Tokens are added to the bucket at a constant rate. If the bucket is empty, the server refuses the request (or forces the client to wait).

***Bucket Capacity ($B$):** Allows for short bursts of traffic.
***Refill Rate ($R$):** The long-term sustainable throughput of your system.

If your storage engine can handle 1,000 requests per second, you set $R=1000$. If you get a sudden spike, the $B$ tokens allow you to absorb it, provided the average stays within $R$.

### Architectural Integration
We place the Token Bucket directly in the `RequestProcessor` pipeline, just after the `SocketChannel` reads the bytes but before the `StorageEngine` executes the command.

[DIAGRAM: Client -> Socket Buffer -> [Token Bucket] -> Storage Engine]
*Annotation: If Bucket == 0, return 429 Too Many Requests or close the connection to signal backpressure.*

### Conceptual Implementation
You don’t need a complex library for this. A simple `AtomicLong` and a timestamp check suffice for a single-threaded reactor:

“`java
public class TokenBucket {
private final long capacity;
private final long refillRatePerNanos;
private long tokens;
private long lastRefillTimestamp;

public synchronized boolean tryConsume() {
refill();
if (tokens > 0) {
tokens–;
return true;
}
return false;
}

private void refill() {
long now = System.nanoTime();
long delta = now – lastRefillTimestamp;
tokens = Math.min(capacity, tokens + (delta / refillRatePerNanos));
lastRefillTimestamp = now;
}
}
“`

### Trade-offs: Why not a Semaphore?
You might be tempted to use a `java.util.concurrent.Semaphore`. A semaphore is great for limiting *concurrency* (how many requests are running *right now*). A Token Bucket limits *rate* (how many requests per second). In storage engines, we care about rate because disk I/O latency is throughput-dependent. If you use a semaphore, you might allow a burst of 100 requests that all hit the disk simultaneously, causing IOPS saturation and latency spikes.

### The Failure Demo
Today, you will use a client script to blast 5,000 requests per second at a server configured to handle only 500. Without the bucket, your JVM heap will grow until an `OutOfMemoryError` or the latency will climb into the seconds as the GC struggles to clean up the queue of pending objects. With the bucket, you will observe the server rejecting requests, keeping the system stable at the defined 500 RPS.

### Looking Ahead
Tomorrow, in Day 27, we will look at how the JVM JIT compiler decides to inline our `tryConsume()` method. Because we are now doing math in the hot path of every request, we need to ensure our Token Bucket doesn’t become a performance bottleneck itself.

—

### Assignment
1.**Integrate:** Add the `TokenBucket` class to your `astra.net` package.
2.**Instrument:** Modify your `NioServer` to instantiate a bucket with a capacity of 100 and a rate of 500 RPS.
3.**Reject:** If `tryConsume()` returns false, send a custom “429 Too Many Requests” response back to the client.
4.**Verify:** Run the load test provided in the repo. Ensure that while the client reports failures, the server process remains responsive and memory usage stays flat.

### Solution Hints
*Don’t use `Thread.sleep` to refill; use the delta-time approach shown in the snippet.
*Your `NioServer` loop should check the bucket before calling `storageEngine.execute()`.
*If you see “Connection Reset” errors, your client might be closing too fast; ensure you handle the server’s response even when it’s a 429.

Page Update

Page Update

System Design Course

System Design Course

Digital Book

100 questions, 40 checklist items. Now rebuild:

Ran 2 commands, read a file

Ran 2 commands, read a file

The Interview Prep Bundle PDF is complete. Here’s exactly what’s in it:

51 pages · 126 KB

Every answer is sourced directly from the StreamSocial and Uber-Lite curriculum content — nothing generic, nothing fabricated. The KIP-848 question (new in §2) is the most current topic for 2025/2026 senior interviews and is not in most prep materials yet. That’s the edge this bundle has. Price it at $79 on Gumroad and it’s ready to upload.

Digital Book

About Distributed Systems with .NET

This PDF is the lead magnet for Distributed Systems with .NET — a 45-post Substack series building one complete distributed log monitoring system in C# on Windows, using .NET Aspire, Azure Service Bus, Azure Event Hubs, Application Insights, Azure Data Explorer, DAPR, and Bicep.

Every Azure service runs on an official free Microsoft local emulator. No Azure subscription is required for any of the 45 posts.

Newsletter Issue

A system design interview tests your ability to design scalable, reliable, and efficient systems that solve real-world problems. Unlike algorithm-focused coding interviews, these assessments evaluate architectural thinking and engineering judgment — and having a repeatable framework is what separates a confident answer from a rambling one.

Why system design Interviews Matter

For senior roles especially, these interviews carry real weight. They demonstrate:

• Technical decision-making with real business implications
• How you manage ambiguity and gather requirements
• Communication ability when discussing complex technical concepts
• Practical experience with large-scale distributed systems

The 5-Step Interview Framework

Step 1: Clarify Requirements (2-3 minutes)

• Ask questions to understand scope and constraints
• Identify users and primary use cases
• Establish scale metrics (users, traffic, data volume)
• Define both functional and non-functional requirements

Step 2: High-Level Design (5-10 minutes)

• Sketch core components (clients, servers, databases, caches)
• Define API contracts between components
• Outline data models and relationships
• Create a basic system flow for primary use cases

Step 3: Deep Dive Into Components (10-15 minutes)

• Select the most critical components to explore in depth
• Address potential bottlenecks
• Explain your technology choices
• Weigh tradeoffs between different approaches

Step 4: Scaling & Optimization (5-10 minutes)

• Identify where the design breaks down at scale
• Apply relevant patterns — sharding, replication, caching
• Address edge cases and potential failures
• Consider performance optimizations

Step 5: Wrap-Up (2-3 minutes)

• Summarize your design
• Acknowledge its limitations
• Suggest future improvements
• Show you understand how the system would evolve

Common Pitfalls to Avoid

• Starting with excessive detail instead of beginning broad
• Proposing solutions before understanding the requirements
• Concentrating on just one aspect (e.g. only the database)
• Overlooking scale considerations from the beginning
• Failing to verbalize your thinking process out loud

Walkthrough: Designing a URL Shortener

Here’s the framework applied to a classic interview question:

1. Requirements: Create short URLs from long ones, redirect users to the originals, scale to millions of URLs, and keep response times fast.
2. High-Level Design: Web servers for creation and redirection, a database for mappings, and a hashing service to generate short codes.
3. Deep Dive: Compare encoding algorithms (Base62 vs. MD5+Base62), design the database schema and indexing, add caching for popular URLs.
4. Scaling: caching and read replicas for the read-heavy workload, database sharding by short code, a CDN for global reach.
5. Wrap-Up: The system handles millions of URLs with sub-100ms response times; future enhancements could include analytics, custom URLs, and expiration policies.

Practice this framework against a handful of real interview questions — URL shortener, rate limiter, chat system, news feed — and it stops being a script you’re reciting and starts being how you actually think.

Want to go deeper on each of these components with hands-on courses and full lesson walkthroughs? Browse our full course catalog, or read more system design breakdowns on our Substack newsletter.