Lesson 21: Advanced Processing Integrations

Hands On System Design with Distributed Systems Implementation ~8 min read 60 min hands-on

InsightStream Analytics Hub: Unified Analytics Command Center

Production log platforms do more than store lines of text. Engineers need sub-second search, periodic rollups, anomaly signals, real-time pattern detection, compact metrics tables, BI-ready exports, and safe natural-language queries — often from different tools that never talk to each other. InsightStream Analytics Hub composes seven analytics engines behind one FastAPI orchestrator on port 8210: ingest a log once, run it through indexing, batch refresh, ML scoring, stream CEP, metrics extraction, BI cache update, and NLP-backed query generation, with every stage visible on a single React dashboard over HTTP and WebSocket.

Core Components

Search Index Engine

Role: Provide fast full-text and structured search over ingested logs without scanning the full table.

SearchEngine in services/search/src/engine.py stores rows in SQLite and mirrors message, service, and level into an FTS5 virtual table. index_log() writes both tables; search() combines FTS MATCH with optional level, service, and time-window filters. This is the retrieval layer every downstream analytics stage reads from.

Batch Analytics Engine

Role: Compute service-level error rates and endpoint latency percentiles across the indexed log corpus.

BatchEngine in services/batch/src/engine.py groups logs by service and hour, derives error_rate_pct, and calculates p50/p95 response times per endpoint. Results land in error_rates, performance, and top_endpoints arrays. The orchestrator triggers a refresh every ten ingests or three seconds, whichever comes first.

ML Scoring Engine

Role: Score each log for latency anomalies and near-term failure risk using rolling statistical baselines.

MLEngine in services/ml/src/engine.py maintains a 200-sample response-time window and a 60-event error deque. A z-score above 2.5 flags is_anomaly; recent error density feeds failure_probability. No external model server — the contract is fast, explainable scoring on the hot path.

Stream CEP Engine

Role: Detect windowed patterns — auth failure spikes, latency degradation, cascading errors, and 5xx bursts — as logs arrive.

StreamEngine in services/stream/src/engine.py loads detector thresholds from config/stream.yaml and tracks per-key sliding windows via WindowState. Matching events emit StreamAlert records with pattern name, severity, and metadata. Alerts accumulate in memory (capped at 200) and surface on the dashboard and /api/stream/alerts.

Time-Series Metrics Engine

Role: Extract compact metric points from raw logs into queryable SQLite tables for dashboards and BI.

MetricsEngine in services/metrics/src/engine.py writes to four tables — http_response, http_status, resource_usage, and throughput — on every ingest. summary() returns rolling averages, error rate, throughput, and per-service breakdowns without re-scanning unstructured log text.

BI Export Engine

Role: Gate exports behind OAuth-style JWT tokens and produce stable CSV files for Tableau or Power BI.

BIEngine in services/bi/src/engine.py authenticates users from config/bi.yaml, caches the latest metrics and batch snapshots, and generates insightstream_metrics_YYYYMMDD_HHMMSS_<id>.csv files under data/exports/. Optional service_filter narrows rows at generation time. Manifest entries power /api/v1/exports/manifest and download routes.

NLP Query Engine

Role: Parse plain-English questions into intents, generate safe SQL strings, and execute them against in-memory log data.

NLPEngine in services/nlp/src/engine.py classifies queries into search, count, aggregate, compare, or investigate via regex patterns, extracts entities (level, services, time window), and runs a lightweight in-memory executor. The goal is bounded, safe answers — not open-ended LLM magic.

Keep going with the full lesson

The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.

Already a member? Sign in