Lesson 7: Distributed Log Analytics — “Turn raw logs into decisions”

Hands On System Design with Distributed Systems Implementation ~7 min read 60 min hands-on

Week 7: Distributed Log Analytics — "Turn raw logs into decisions"

Part of the 254-Day Hands-On System Design series. Style reference: Day 165: Real-Time SLA Monitoring.

What we're building today

Shopify processes billions of checkout events. Datadog turns them into error spikes you see in seconds—not after a nightly batch job. This week you wire seven analytics layers into one pipeline: batch MapReduce, tumbling windows, sliding averages, user sessions, anomaly scoring, alert rules, and a live React dashboard.

By the end you'll have:

  • MapReduce word-count and pattern-frequency jobs on real log batches

  • 60-second tumbling windows with error-rate aggregation

  • 30-second sliding windows for response-time trends

  • Session tracking with engagement scoring

  • Three-method anomaly ensemble (z-score, temporal, isolation heuristic)

  • Threshold-based alerts with deduplication

  • A WebSocket dashboard showing all metrics live

Why this matters

  • Google's MapReduce paper (2004) made petabyte log analysis possible—Hadoop and Spark descend directly from it.

  • Flink and Kafka Streams use windowing primitives almost identical to what you build on Day 46–47.

  • Stripe sessionizes checkout flows to detect fraud rings; your sessionizer uses the same timeout-and-group pattern.

  • Cloudflare anomaly detection on request logs prevents DDoS false negatives—ensemble voting beats any single detector.

Batch answers "what happened yesterday?" Streaming windows answer "what's happening now?" Alerts close the loop.

Architecture

Everything runs inside a single FastAPI service on port 8090. A background log generator feeds synthetic JSON events into the pipeline. SQLite stores logs, alerts, and MapReduce results locally—no external database required for this demo.

Project layout

Each file maps to one lesson from Days 45–51. The pipeline module wires them together so every log event passes through all processors in parallel.

Design choices worth knowing

  • Fan-out pipeline — one ingest path, many processors. Same idea as Apache Beam multi-output transforms and Flink operator chains.

  • MapReduce without Hadoop — ProcessPoolExecutor plus shuffle-by-key gives you map-reduce semantics on a laptop. Production systems use Spark or Dataproc for the same pattern at scale.

  • Tumbling vs sliding — tumbling windows answer "how many errors in this minute?" Sliding windows answer "what's the current trend?" Both show up in Datadog and Prometheus.

  • Session gap timeout — a 30-minute inactivity boundary groups events into user journeys, matching how Google Analytics defines sessions.

  • Ensemble anomaly detection — requiring 2-of-3 methods to agree cuts false positives. Datadog uses a similar voting pattern.

  • React via CDN — no npm build step. Babel compiles JSX in the browser so you can focus on the analytics code.

Data flow

Each log event fans out to all processors in parallel—same pattern as Apache Beam's multi-output transforms.

What happens to each event

  1. The log generator (or a manual POST /api/logs) emits a JSON event into the pipeline.

  2. The event fans out in parallel to every processor:

Keep going with the full lesson

The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.

Already a member? Sign in