Lesson 7: Notification Dispatch Control Plane

What We Are Building

InfraWatch Notify — a production notification dispatch console modeled after PagerDuty and Opsgenie:

1. Multi-channel delivery — Email, SMS, Slack, webhook, and push providers behind a unified interface

2. Template engine — Jinja2 rendering with per-channel formats and preview

3. Delivery pipeline — Redis priority queue, background worker, retry with backoff, rate limiting

4. Preference routing — Severity-based channel selection, quiet hours, suppression rules

5. Reliability layer — Circuit breakers, delivery tracking timeline, integration test harness

6. Operator console — Dark-theme dashboard with live WebSocket feed and channel health

Core Concept: Alerts Are Inputs, Notifications Are Outputs

PagerDuty does not blast every engineer for every metric spike. It routes alert events through user preferences, escalation policies, and channel capacity before a human ever sees a ping. The non-obvious insight: notification delivery is its own distributed system — separate from alert detection, separate from metric storage.

A firing alert is a signal. A notification is a delivery attempt with its own lifecycle: queued → processing → delivered or failed → retry. Datadog and Grafana solve the "what broke" problem; Notify solves "who gets told, how, and did it actually arrive."

Where This Fits

Notify sits downstream of alert systems (like InfraWatch Alerts). It assumes alerts arrive with severity, service context, and a message — then handles the hard part of reliable multi-channel fan-out.

Component Architecture

Orchestrator — Receives alerts, resolves recipients, renders templates, enqueues delivery jobs.
Preference Router — Applies quiet hours, severity-channel matrix, and global opt-out.
Queue Manager — Redis-backed priority heap; CRITICAL jobs dequeue before LOW.
Delivery Worker — Background thread processes batches, records events, handles retries.
Channel Providers — Pluggable senders with validation and latency tracking.
WebSocket Hub — Pushes delivery state changes to the operator console.

Control & Data Flow

Alert POST → preference check per user → template render per channel → idempotency key → enqueue → worker dequeue → circuit breaker check → channel send → delivery event logged → WebSocket broadcast → dashboard card refresh.

Escalation worker polls unacknowledged alerts. After the configured timeout, it bumps escalation level and re-triggers fan-out to the next tier — exactly how Opsgenie tiered paging works.

When you trace a single alert through the code, this is the path it follows:

1. Operator or API POSTs alert to /api/v1/alerts/.

2. Orchestrator loads all user preferences.

3. PreferenceService.should_send() checks global enable, quiet hours, and severity exceptions.

4. Channels resolved per severity (user override or defaults).

5. Jinja2 template rendered per channel (alert_email, alert_sms, etc.).

6. Idempotency key prevents duplicate delivery for same alert + user + channel.

7. Notification enqueued in Redis priority queue.

8. Delivery worker dequeues, checks circuit breaker and rate limit, sends via channel provider.

9. Delivery events logged; WebSocket broadcasts status to dashboard.

Notification State Machine

Each notification record tracks its own state independently of the parent alert.

The parent alert moves through its own states separately:

Design Notes Worth Remembering

These patterns show up in real on-call systems. You implemented smaller versions of each one: