Lesson 21: Advanced Processing Integrations
InsightStream Analytics Hub: Unified Analytics Command Center
Production log platforms do more than store lines of text. Engineers need sub-second search, periodic rollups, anomaly signals, real-time pattern detection, compact metrics tables, BI-ready exports, and safe natural-language queries — often from different tools that never talk to each other. InsightStream Analytics Hub composes seven analytics engines behind one FastAPI orchestrator on port 8210: ingest a log once, run it through indexing, batch refresh, ML scoring, stream CEP, metrics extraction, BI cache update, and NLP-backed query generation, with every stage visible on a single React dashboard over HTTP and WebSocket.
Core Components
Search Index Engine
Role: Provide fast full-text and structured search over ingested logs without scanning the full table.
SearchEngine in services/search/src/engine.py stores rows in SQLite and mirrors message, service, and level into an FTS5 virtual table. index_log() writes both tables; search() combines FTS MATCH with optional level, service, and time-window filters. This is the retrieval layer every downstream analytics stage reads from.
Batch Analytics Engine
Role: Compute service-level error rates and endpoint latency percentiles across the indexed log corpus.
BatchEngine in services/batch/src/engine.py groups logs by service and hour, derives error_rate_pct, and calculates p50/p95 response times per endpoint. Results land in error_rates, performance, and top_endpoints arrays. The orchestrator triggers a refresh every ten ingests or three seconds, whichever comes first.
ML Scoring Engine
Role: Score each log for latency anomalies and near-term failure risk using rolling statistical baselines.
MLEngine in services/ml/src/engine.py maintains a 200-sample response-time window and a 60-event error deque. A z-score above 2.5 flags is_anomaly; recent error density feeds failure_probability. No external model server — the contract is fast, explainable scoring on the hot path.
Stream CEP Engine
Role: Detect windowed patterns — auth failure spikes, latency degradation, cascading errors, and 5xx bursts — as logs arrive.
StreamEngine in services/stream/src/engine.py loads detector thresholds from config/stream.yaml and tracks per-key sliding windows via WindowState. Matching events emit StreamAlert records with pattern name, severity, and metadata. Alerts accumulate in memory (capped at 200) and surface on the dashboard and /api/stream/alerts.
Time-Series Metrics Engine
Role: Extract compact metric points from raw logs into queryable SQLite tables for dashboards and BI.
MetricsEngine in services/metrics/src/engine.py writes to four tables — http_response, http_status, resource_usage, and throughput — on every ingest. summary() returns rolling averages, error rate, throughput, and per-service breakdowns without re-scanning unstructured log text.
BI Export Engine
Role: Gate exports behind OAuth-style JWT tokens and produce stable CSV files for Tableau or Power BI.
BIEngine in services/bi/src/engine.py authenticates users from config/bi.yaml, caches the latest metrics and batch snapshots, and generates insightstream_metrics_YYYYMMDD_HHMMSS_<id>.csv files under data/exports/. Optional service_filter narrows rows at generation time. Manifest entries power /api/v1/exports/manifest and download routes.
NLP Query Engine
Role: Parse plain-English questions into intents, generate safe SQL strings, and execute them against in-memory log data.
NLPEngine in services/nlp/src/engine.py classifies queries into search, count, aggregate, compare, or investigate via regex patterns, extracts entities (level, services, time window), and runs a lightweight in-memory executor. The goal is bounded, safe answers — not open-ended LLM magic.
Keep going with the full lesson
The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.
Already a member? Sign in