Day 9: Query Key-Value Metadata Using the Streams API to Filter and Aggregate Records

Day 9: Query Key-Value Metadata Using the Streams API to Filter and Aggregate Records

In Day 8, you implemented the CommandRouter, using Java 21 pattern matching to route incoming StorageCommand records to the appropriate persistence handlers. That structure is excellent for point-lookups—finding one key at a time. Today, we shift from "find me this specific record" to "tell me about the state of my storage engine." We are building a metadata query layer that allows you to inspect the health and distribution of your data without scanning every byte on disk.

The Problem: The "Full Scan" Trap

In a distributed system like Apache Cassandra or Google Spanner, the SSTable (Sorted String Table) files can grow to many gigabytes. If you need to know how many keys have a specific prefix, or the average size of values in a specific range, you cannot simply read the entire file into memory. You would trigger an OutOfMemoryError (OOM) before you finished the first file.

This is the "Full Scan" trap. When an operator asks for metadata—like "count all keys modified in the last hour"—naive implementations force the system to deserialize every record. This spikes CPU usage, blocks the storage engine from processing writes, and often crashes the node under load.

The Concept: Lazy Projection and Aggregation

To avoid the OOM trap, we use Lazy Projection. Instead of loading full Record objects, we stream the file metadata, extract only the necessary fields (like key or timestamp), and perform calculations in a single pass using the Java Streams API.