Lesson 18: Log Source Integration
SourceBridge: Unified Log Source Hub
Your payment API throws errors at 2 AM. Logs might sit in Linux syslog, a Windows Security channel, AWS Lambda CloudWatch, Azure AppTraces, a GKE pod, or a Postgres audit table. Production teams do not pick one source — they fan in all of them. SourceBridge is a runnable hub that composes seven collector engines behind one FastAPI service on port 8300, normalizes every event to a single schema, stores them in SQLite, and pushes live KPI updates over WebSocket to a React dashboard.
Core Components
Linux File Collector
Role: Tail syslog and webserver log files with byte-offset tracking and batch discovery.
LinuxCollectorEngine in services/linux/src/engine.py watches data/log-sources/linux/*.log, reads new lines since the last offset, deduplicates by content hash, and classifies entries as system, webserver, or auth based on filename and content. normalize() maps raw lines to the unified record shape with hostname metadata from os.uname().
Windows Event Collector
Role: Emit System, Application, and Security channel events for cross-platform pipelines.
WindowsEventEngine in services/windows/src/engine.py generates realistic Windows event records (event IDs 4624, 4625, 7036, and others) on Linux hosts where native WinEvt APIs are unavailable. Each batch covers all three channels; normalize() maps Information/Warning/Error levels to info, warning, and error severities.
CloudWatch Collector
Role: Pull log group streams from local JSONL files with SQLite checkpoint tokens.
CloudWatchEngine in services/cloudwatch/src/engine.py mirrors AWS CloudWatch Logs semantics: inject_logs() writes Lambda-style JSONL under data/cloudwatch/log_groups/, and collect_batch() resumes from per-stream offsets stored in the cw_checkpoints table. Without checkpointing, replaying a stream double-counts invocations.
Azure Monitor Collector
Role: Run demo-mode KQL-style workspace queries against AppTraces, SecurityEvent, and Perf tables.
AzureMonitorEngine in services/azure/src/engine.py operates in demo mode (configured in config/sources.yaml) and emits workspace-scoped telemetry samples. normalize() preserves table name, resource name, and subscription ID in metadata for downstream filtering.
GCP Logging Collector
Role: Tail multi-project JSONL log files with per-project line offsets.
GCPLoggingEngine in services/gcp/src/engine.py reads data/gcp/{project}.jsonl for each project in config (default proj-payments and proj-analytics). Checkpoints live in the gcp_checkpoints table; seed_project_logs() appends structured entries with insert_id for idempotency testing.
Container Log Collector
Role: Stream Docker stdout with severity parsing and container metadata enrichment.
ContainerLogEngine in services/container/src/engine.py calls docker ps and docker logs when the socket is available, enriching each line with container_name, container_id, parsed log level, and optional JSON structure. Without Docker, it falls back to data/containers/container.log. Raw stdout without container context is nearly useless in a large cluster.
Database Audit Collector
Role: Parse normalized audit rows and run inline threat detection.
DatabaseAuditEngine in services/database/src/engine.py reads uncollected rows from audit_raw, analyzes operations in _analyze() for bulk DELETE on users tables and failed logins, records matches in audit_threats, and increments threats_detected. seed_demo_rows() inserts fresh audit trails for demo and test runs.
Platform Orchestrator
Role: Wire engines, run background polling, assemble dashboard payloads, and expose Prometheus counters.
Keep going with the full lesson
The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.
Already a member? Sign in