Lesson 23: Security Log Processing

Hands On System Design with Distributed Systems Implementation ~10 min read 60 min hands-on

LogShield SOC: Security Operations Command Center

Security teams cannot read millions of log lines by hand. They correlate related events into incidents, match signatures and threat intelligence, score behavioral anomalies, execute automated response playbooks, preserve tamper-evident compliance evidence, and analyze network flows — all in parallel. LogShield SOC unifies those seven detection and response layers behind one FastAPI orchestrator on port 8230. Trigger a simulation on the dashboard and the operations feed on the right shows exactly what each stage did downstream.

Core Components

SIEM Correlation Engine

Role: Normalize raw events, score risk, and group related authentication failures into incidents.

SIEMEngine in services/siem/src/engine.py uses EventNormalizer to classify payloads (auth failures, privilege escalation, web attacks), RiskScorer to weight severity by event type and metadata multipliers, and a sliding window keyed by user:source_ip to open a brute-force incident when five failures arrive within sixty seconds (configurable in config/siem.yaml). Incidents surface on /api/incidents and feed compliance evaluation when event_type becomes incident_created.

Threat Rule Engine

Role: Match regex signatures against log payloads with optional rate thresholds.

RuleEngine in services/threat/src/engine.py loads detection_rules from config/rules.yaml — SQL injection, failed-login patterns, and shell command injection. PatternMatcher compiles each rule; thresholded rules (e.g., five hits per source IP in sixty seconds) suppress noise until the window fills. Detections append to an in-memory list exposed via /api/threats.

User Behavior Analytics Engine

Role: Learn per-user login baselines and flag statistical and temporal anomalies.

UBAEngine in services/uba/src/engine.py maintains BaselineManager state for login hour, count, and known IPs per user. Off-hours access (before 6 AM or after 10 PM) combined with a new IP triggers off_hours_new_ip; trained users with login-hour z-scores above 2.5 trigger unusual_login_time. The simulate_anomaly metadata flag forces a high-risk alert for demos. Alerts appear on /api/uba/alerts.

IOC Matcher Engine

Role: Scan event text and source IPs against a built-in threat-intelligence database.

IOCMatcherEngine in services/ioc/src/engine.py extracts IPv4 addresses and domains from payloads, looks them up in IOCDatabase (Tor exit nodes, known C2 servers, malware distribution hosts), and records IOCMatch entries. A single hit upgrades an event from suspicious to confirmed threat and can trigger the c2_communication_response playbook via the ioc_match trigger.

Incident Response Playbook Engine

Role: Execute YAML-defined SOAR playbooks when automated triggers fire.

PlaybookEngine in services/response/src/engine.py loads templates from config/playbooks/ (brute_force_response, c2_communication_response, malware_response, network_scan_response, data_exfiltration_response). auto_respond() matches a trigger string to playbook triggers, resolves ${variable} placeholders from context, and runs simulated handlers (block_ip, isolate_host, preserve_logs, send_slack_alert, etc.). Executions and audit entries surface on /api/responses.

Compliance Evidence Engine

Role: Map security events to PCI-DSS and SOC2 controls and maintain a hash-chained evidence ledger.

ComplianceRuleEngine in services/compliance/src/engine.py evaluates events against criteria in config/compliance.yaml. Matching events produce EvidenceEntry records where each SHA-256 hash chains to the previous entry (prev_hash → hash). coverage() reports per-framework met requirements, gaps, and overall_coverage_pct. Chain integrity is verifiable via verify_chain().

Network Traffic Analyzer

Role: Parse syslog connection lines, track flow metrics, and detect port scans.

Keep going with the full lesson

The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.

Already a member? Sign in