Lesson 11: Performance Optimization — Make every log byte earn its keep
What we're building today
When Cloudflare ingests 10M+ HTTP logs per second, they don't write one event at a time—they batch, compress, and skip duplicates before touching disk. Datadog caches hot queries in memory tiers. ClickHouse picks columnar storage because analytics scans columns, not rows. This lesson wires seven performance layers into one platform.
By the end you'll have:
Pipeline profiler with bottleneck detection
Self-tuning adaptive batching
L1 LRU query cache
Row / columnar / hybrid storage selection
Bloom filter deduplication
Delta encoding with zlib compression
Adaptive worker scaling
Why this matters
Kafka producers batch messages because each network round-trip costs milliseconds—your adaptive batcher learns the sweet spot automatically.
Redis sits in front of Postgres for hot keys—your L1 cache does the same for repeated log queries.
S3 lifecycle rules move cold data to cheaper tiers—your storage optimizer picks row vs columnar based on access patterns.
Cassandra uses bloom filters to skip SSTable reads—your bloom layer skips duplicate log fingerprints at ingest.
InfluxDB delta-encodes timestamps—your encoder stores only what changed since the last baseline.
Performance is not one trick—it is a stack of small decisions that compound.
Architecture
Single FastAPI service on port 8095. All performance modules run in-process with SQLite persistence.
How data moves through the system
Ingest path: Log event enters batcher. When the batch fills, the bloom filter checks the fingerprint. New logs get delta-encoded. The storage optimizer writes in the best format. The profiler records stage timing.
Query path: API checks L1 cache. On miss, it reads SQLite and caches the result. A second identical query hits cache—watch the hit rate climb on the dashboard.
Scale path: Allocator polls CPU and queue depth every 3 seconds. Scale up above 70% CPU or 500 pending items. Scale down when idle.
Control flow detail:
Ingest —
AdaptiveBatcher.add()→ on flush, bloom check → delta encode → storage write → SQLite persist.Query — L1 cache lookup → on miss, SQLite read → cache populate → profiler records query stage.
Profiler — background psutil loop + per-stage timing → bottleneck detection.
Allocator — polls CPU and batch queue depth → scale workers with cooldown.
WebSocket
/wsbroadcasts platform stats every 2 seconds.Background generator — daemon thread ingests sample logs every 1.2s for live dashboard metrics.
Project layout
Open lesson-11_Performance_Optimization_Platform/:
Design highlights
No npm build: React dashboard loads from CDN—zero frontend toolchain.
Pure-Python bloom filter: Uses SHA-256 hashing—no extra hashing libraries required.
Delta + zlib: Stdlib compression keeps dependencies minimal.
Storage without Parquet: Columnar mode groups fields into JSON column maps—teaches the concept without heavy dependencies.
SQLite + threading lock: API and background log generator share one database safely.
Background generator: Daemon thread feeds logs every 1.2s so dashboard metrics move without clicks.
Optimization score: Composite metric blends cache hit rate, compression savings, throughput, and worker utilization.
PEP 668 safe scripts: Shell scripts use
.venv/bin/python -m pipexplicitly—works on Debian externally-managed Python.No API keys: No external credentials required;
DATA_DIRis the only runtime config.
API reference
Base URL: http://localhost:8095 — interactive docs at /docs
Keep going with the full lesson
The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.
Already a member? Sign in