Lesson 6 : Alert Management Control Plane

Hands On FullStack Development Course with Infrastructure Management Product implementation ~9 min read 60 min hands-on

Lesson 6 : Alert Management Control Plane

What We Are Building

InfraWatch Alerts — a production alert management console modeled after PagerDuty and Datadog:

  1. Rule engine — Threshold, anomaly, and expression-based rules with templates

  2. Evaluation pipeline — Sustained breach detection, deduplication, suppression windows

  3. Lifecycle processing — Severity scoring, escalation policies, auto-resolution

  4. Query plane — Search, statistics, bulk operations, CSV/JSON export

  5. Operator console — Dark-theme dashboard with live WebSocket updates

  6. Notification routing — Channel delivery with circuit-breaker protection

Core Concept: Signal vs Noise

Every monitoring platform ingests thousands of metric points per minute. The hard problem is not detection — it is deciding which signals deserve human attention. Datadog and PagerDuty solve this with pending timers (avoid flapping), fingerprint deduplication (one alert per root cause), and suppression windows (maintenance silence).

The non-obvious insight: alert state is a first-class entity, not a side effect of rule evaluation. A rule fires; an alert instance is born with its own lifecycle. Operators acknowledge, escalate, and resolve alerts — not rules.

Where This Fits

Code

Alerts sit between completed metric pipelines and human response. They assume metrics arrive reliably and workers can evaluate rules on schedule.

Component Architecture

Ingest — Agents POST metric samples; each sample triggers rule evaluation.
Evaluator — Compares values against thresholds or Z-score anomaly models.
Processor — Manages pending→firing transitions, severity scoring, history audit.
Query API — Powers search, statistics, and export for the operator console.
WebSocket — Pushes state changes to the dashboard without polling.

Control & Data Flow

Metric arrives → stored in PostgreSQL → evaluator checks all enabled rules → suppression filter → dedup fingerprint → create or update alert instance → pending timer → firing → notification router → WebSocket broadcast → dashboard card refresh.

Step-by-step path your code follows:

  1. Agent or simulator POSTs a sample to /api/v1/metrics/ingest.

  2. Sample lands in the metric_samples table.

  3. The evaluation engine loads every enabled rule.

  4. The suppression service checks maintenance windows and regex patterns.

  5. The evaluator compares the value against a threshold or Z-score anomaly model.

  6. An alert instance is created or updated — pending becomes firing after for_duration_seconds.

  7. The notification router logs delivery; WebSocket pushes the update.

  8. Dashboard stats cards and the alert table refresh.

Alert Lifecycle

States: OK → Pending → Firing → Acknowledged → Resolved, with a Suppressed branch during maintenance windows. Pending requires sustained breach (for_duration_seconds) before firing — the same pattern Prometheus for: clauses use.

Real-World Context

  • PagerDuty routes firing alerts through escalation policies with time-delayed notifications.

  • Datadog monitors use evaluation windows to prevent alert storms from brief spikes.

  • Grafana alert rules separate rule definition from alert instance management.

Your console implements all three patterns in one self-contained stack.

Design Notes

These choices show up directly in the code you will run:

  • Pending before firing — Rules use for_duration_seconds so a one-second CPU spike does not wake anyone at 3 a.m.

  • Fingerprint dedup — A SHA-256 fingerprint per rule and metric stops duplicate alert rows from flooding the table.

  • DB-first state — Every transition is written to alert_state_history so you can audit what happened and when.

  • Same-origin WebSocket — The frontend connects to ws://<host>/ws/alerts; Vite and nginx proxy it correctly in both local and Docker modes.

  • Self-contained project — Clone infrawatch-alerts/ alone. Everything you need is inside that folder.

  • No API keys in source — Database credentials are local Docker defaults for development. Set SECRET_KEY in .env before production.

Project Layout

Code

Success Criteria

After completing this lesson you should be able to:

Keep going with the full lesson

The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.

Unlock the full lesson Create free account

Already a member? Sign in