Lesson 11: Performance Optimization — Make every log byte earn its keep

Hands On System Design with Distributed Systems Implementation ~8 min read 60 min hands-on

What we're building today

When Cloudflare ingests 10M+ HTTP logs per second, they don't write one event at a time—they batch, compress, and skip duplicates before touching disk. Datadog caches hot queries in memory tiers. ClickHouse picks columnar storage because analytics scans columns, not rows. This lesson wires seven performance layers into one platform.

By the end you'll have:

  • Pipeline profiler with bottleneck detection

  • Self-tuning adaptive batching

  • L1 LRU query cache

  • Row / columnar / hybrid storage selection

  • Bloom filter deduplication

  • Delta encoding with zlib compression

  • Adaptive worker scaling

Why this matters

  • Kafka producers batch messages because each network round-trip costs milliseconds—your adaptive batcher learns the sweet spot automatically.

  • Redis sits in front of Postgres for hot keys—your L1 cache does the same for repeated log queries.

  • S3 lifecycle rules move cold data to cheaper tiers—your storage optimizer picks row vs columnar based on access patterns.

  • Cassandra uses bloom filters to skip SSTable reads—your bloom layer skips duplicate log fingerprints at ingest.

  • InfluxDB delta-encodes timestamps—your encoder stores only what changed since the last baseline.

Performance is not one trick—it is a stack of small decisions that compound.

Architecture

Single FastAPI service on port 8095. All performance modules run in-process with SQLite persistence.

How data moves through the system

Ingest path: Log event enters batcher. When the batch fills, the bloom filter checks the fingerprint. New logs get delta-encoded. The storage optimizer writes in the best format. The profiler records stage timing.

Query path: API checks L1 cache. On miss, it reads SQLite and caches the result. A second identical query hits cache—watch the hit rate climb on the dashboard.

Scale path: Allocator polls CPU and queue depth every 3 seconds. Scale up above 70% CPU or 500 pending items. Scale down when idle.

Control flow detail:

  1. Ingest — AdaptiveBatcher.add() → on flush, bloom check → delta encode → storage write → SQLite persist.

  2. Query — L1 cache lookup → on miss, SQLite read → cache populate → profiler records query stage.

  3. Profiler — background psutil loop + per-stage timing → bottleneck detection.

  4. Allocator — polls CPU and batch queue depth → scale workers with cooldown.

  5. WebSocket /ws broadcasts platform stats every 2 seconds.

  6. Background generator — daemon thread ingests sample logs every 1.2s for live dashboard metrics.

Project layout

Open lesson-11_Performance_Optimization_Platform/:

Design highlights

  • No npm build: React dashboard loads from CDN—zero frontend toolchain.

  • Pure-Python bloom filter: Uses SHA-256 hashing—no extra hashing libraries required.

  • Delta + zlib: Stdlib compression keeps dependencies minimal.

  • Storage without Parquet: Columnar mode groups fields into JSON column maps—teaches the concept without heavy dependencies.

  • SQLite + threading lock: API and background log generator share one database safely.

  • Background generator: Daemon thread feeds logs every 1.2s so dashboard metrics move without clicks.

  • Optimization score: Composite metric blends cache hit rate, compression savings, throughput, and worker utilization.

  • PEP 668 safe scripts: Shell scripts use .venv/bin/python -m pip explicitly—works on Debian externally-managed Python.

  • No API keys: No external credentials required; DATA_DIR is the only runtime config.

API reference

Base URL: http://localhost:8095 — interactive docs at /docs

Keep going with the full lesson

The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.

Already a member? Sign in