Lesson 11: Performance Optimization

Adaptive Log Performance: A Pipeline That Measures and Tunes Itself

Summary

This article walks through a log ingestion and query platform built around one idea: every performance technique should report its own effect, and the techniques that can be driven by a live signal should be. The system combines a profiled single-pass ingest path, load-aware batch sizing, a two-tier query cache, row and columnar storage formats, bloom filters, delta encoding and backlog-driven worker allocation. It runs alone with an embedded database or across Kafka, Redis and PostgreSQL.

Core Concepts

Profiling with a baseline. An optimisation without a comparison is an opinion. A deliberately naive ingest path (JSON tree and one write per event) runs next to the real one (streaming tokens, one write per batch) on the same input, and throughput, allocation and write calls are reported side by side.

Adaptive batching. A load score weights CPU, heap, backlog and batch latency. High load shrinks batches so each finishes quickly; low load grows them to amortise fixed costs. A dead zone between the thresholds prevents oscillation.

Query caching. Dashboards repeat the same questions. Results are cached in process (Caffeine) and, in distributed mode, in Redis. Keys include a minute bucket, writes mark a service dirty, and invalidations are coalesced every two seconds so a busy service does not evict its cache on every batch.

Storage formats. Row segments make appends cheap; columnar segments make scans cheap. Each format is used where it wins, and a compactor moves data from one to the other when a time window closes.