System Design Roadmap - Hands-On System Design Lessons

- Hands-On Tutorial

system11

Hands-On System Design Tutorial

## Lesson 10 : Advanced Dashboard Platform

## What We Are Building

InfraWatch **Dashboard** — a Grafana-class observability console for infrastructure operators:

1. **Composable Grid** — Drag-and-drop widgets with persistent layouts and a widget library
2. **Advanced Charts** — Multi-series, stacked, scatter, heatmap, latency distribution, status timelines
3. **Customization** — Per-dashboard themes, save/load layouts, tokenized sharing
4. **Interactive Analytics** — Cross-filtering, drill-down hierarchy, time-range presets
5. **performance Layer** — LTTB downsampling, Redis query cache, WebSocket live streams
6. **Template Gallery** — Semver-versioned marketplace with ratings and one-click deployment

—

## Core Concept

Datadog and Grafana do not render raw million-point time series to the browser. They **downsample at query time** (LTTB algorithm), **cache aggregated buckets in Redis**, and **push deltas over WebSockets** so the UI stays at 60fps. The insight most tutorials miss: the dashboard is a *composition engine* — widgets are independent data contracts, not hard-coded charts. Layout state (x, y, w, h) persists separately from widget config, enabling template reuse without duplicating query logic.

—

## Where This Fits

“`
InfraWatch Reports (exports + analytics API)
│
InfraWatch Dashboard ◀ this module
│
Operator fleet monitoring
“`

Dashboard consumes aggregated metric samples upstream and presents the operator’s primary control surface for fleet health.

—

## Component Architecture

“`
React Console (:5230)
│ REST + WebSocket
FastAPI Gateway (:5060)
│
┌────┴────┬──────────┐
PostgreSQL Redis LTTB Engine
:5490 :6440
“`

### Project Layout

Once you open the project folder, here is how the code is organized. Think of it as three layers: the API (backend), the browser UI (frontend), and the scripts that wire everything together.

“`
infrawatch-dashboard/
├── backend/
│ ├── app/
│ │ ├── main.py # FastAPI app, lifespan, CORS
│ │ ├── API/
│ │ │ ├── dashboards.py # Dashboard CRUD, layout, widgets
│ │ │ ├── charts.py # Multi-series, stacked, scatter, heatmap
│ │ │ ├── filters.py # Cross-filter options
│ │ │ ├── drilldown.py # Hierarchy navigation
│ │ │ ├── templates.py # Marketplace CRUD, instantiate
│ │ │ ├── sharing.py # Share tokens
│ │ │ ├── performance.py # Cache stats, WebSocket, Prometheus
│ │ │ └── operations.py # Overview, demo, E2E
│ │ ├── core/ # config, database
│ │ ├── models/ # Dashboard, Widget, Template, MetricSample
│ │ ├── schemas/ # Pydantic request/response
│ │ └── services/
│ │ ├── metrics_service.py # Query, aggregate, LTTB pipeline
│ │ ├── cache_service.py # Redis query cache
│ │ ├── downsample.py # LTTB algorithm
│ │ ├── template_service.py # Semver, instantiate
│ │ └── seed.py # 1900+ metric samples
│ ├── tests/
│ │ ├── conftest.py
│ │ └── test_api.py
│ ├── Dockerfile
│ └── requirements.txt
├── frontend/
│ ├── src/
│ │ ├── pages/ # Console, Marketplace, performance
│ │ ├── components/
│ │ │ ├── dashboard/ # Grid, WidgetRenderer, FilterPanel
│ │ │ └── Layout.tsx
│ │ ├── API/client.ts
│ │ ├── store/filterStore.ts # Zustand state
│ │ └── hooks/useMetricsStream.ts
│ ├── package.json
│ └── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── requirements-dev.txt
├── build.sh
├── stop.sh
└── cleanup.sh
“`

### How the Pieces Work Together

**Widget Grid.** Dashboards store layout as JSON (`x`, `y`, `w`, `h` per widget). `react-grid-layout` handles drag-and-resize on the frontend. Layout saves via `PUT /API/v1/dashboards/{id}/layout` with optimistic versioning — the server returns HTTP 409 on version conflict.

**Chart Pipeline.** `MetricsService.timeseries()` buckets raw `MetricSample` rows into 5-minute windows, then applies LTTB downsampling when points exceed 1920. Results cache in Redis (60s TTL) before JSON serialization.

**Cross-Filter and Drill-Down.** Zustand store holds `timeRange`, `services[]`, `regions[]`, and `drillPath[]`. Progressive filter options come from `/API/v1/filters/available`. Drill-down navigates service → endpoint → region → environment via `/API/v1/drilldown/navigate`.

**Template Marketplace.** Templates store `config: { layout, widgets, theme }`. Instantiating creates a new Dashboard and Widget rows. Config changes bump semver (major if widget removed, minor if added, patch otherwise).

**Real-Time Stream.** WebSocket at `/API/v1/performance/stream` pushes `metrics_batch` every 3 seconds. The frontend `useMetricsStream` hook updates the live point counter in the performance panel.

—

## Control & Data Flow

1. Operator selects time range and cross-filters (service, region)
2. API checks Redis; cache misses query indexed `metric_samples` rows
3. LTTB reduces points to ≤1920 before JSON serialization
4. WebSocket pushes metric batches every 3s for live widgets
5. Drag-resize triggers versioned layout save (optimistic locking on conflict)

—

## Widget Lifecycle

—

## Operator Console

The UI follows a Grafana-inspired dark theme (`#111217` background, `#73BF69` accent). Four alternate themes ship with the project: light, ocean, and sunset.

### Widget Types

—

## API Reference

These are the endpoints you will call from the frontend or test with curl.

### Dashboards

| Endpoint | Method | Purpose |
|———-|——–|———|
| `/API/v1/dashboards` | GET | List dashboards |
| `/API/v1/dashboards` | POST | Create dashboard |
| `/API/v1/dashboards/{id}` | GET | Dashboard detail + widgets |
| `/API/v1/dashboards/{id}/layout` | PUT | Persist grid layout |
| `/API/v1/dashboards/{id}/theme` | PUT | Change theme |
| `/API/v1/dashboards/{id}/widgets` | POST | Add widget |

### Charts

### Filters, Drill-Down, Templates, and Sharing

| Endpoint | Method | Purpose |
|———-|——–|———|
| `/API/v1/filters/available` | GET | Cross-filter options with counts |
| `/API/v1/drilldown/navigate` | POST | Drill into hierarchy level |
| `/API/v1/drilldown/details` | POST | Detail records for path |
| `/API/v1/templates` | GET | Marketplace search |
| `/API/v1/templates/{id}/instantiate` | POST | Deploy template as dashboard |
| `/API/v1/templates/{id}/rate` | POST | Rate template |
| `/API/v1/sharing/{id}/share` | POST | Create share link |
| `/API/v1/sharing/shared/{token}` | GET | View shared dashboard |

### performance and Operations

Quick health check after the stack is running:

“`bash
curl -s http://localhost:5060/API/health
curl -s http://localhost:5060/API/v1/operations/overview | python3 -m json.tool
“`

—

## Build, Test, and Demo

Everything runs from inside the `infrawatch-dashboard` folder. Make the scripts executable once, then pick the mode that fits your machine.

“`bash
cd infrawatch-dashboard
chmod +x build.sh stop.sh cleanup.sh
“`

### Step 1 — Run Tests First

Always start here. If tests pass, your environment is wired correctly before you spin up services.

“`bash
./build.sh test # 14 pytest + 1 vitest
“`

| Suite | Count | What It Covers |
|——-|——-|—————-|
| pytest | 14 | health, overview, charts, filters, drilldown, templates, layout, sharing, cache, e2e |
| vitest | 1 | App renders InfraWatch Dashboard branding |

Backend tests use in-memory SQLite with `StaticPool` and dependency overrides in `tests/conftest.py`.

### Step 2 — Start the Stack (Hybrid Mode)

This is the recommended path for daily development. Postgres and Redis run in Docker; the API and UI run on your host machine.

### Without Docker (local dev)

“`bash
cd infrawatch-dashboard
./build.sh local # Postgres/Redis in Docker, API+UI on host
./build.sh demo # inject metrics, verify non-zero chart points
“`

Open **http://localhost:5230** — Command Center shows live KPI cards and charts.

Expected output from `./build.sh demo`:

“`
{“status”:”healthy”,”service”:”infrawatch-dashboard”}
Metric samples: 1900+ | Chart points: 9+
{“passed”: true}
Open http://localhost:5230
“`

### Step 3 — Full Docker Stack (Optional)

Use this when you want every service inside containers, matching a production-like layout.

### With Docker (full stack)

“`bash
./build.sh docker
./build.sh demo
“`

Requires Docker Hub access for `python:3.12-slim`, `node:22-alpine`, `nginx:alpine`, `postgres:16-alpine`, and `redis:7-alpine`. If your network blocks Docker Hub, pre-pull the images:

“`bash
docker pull python:3.12-slim node:22-alpine nginx:alpine postgres:16-alpine redis:7-alpine
./build.sh docker
“`

Or run compose directly:

“`bash
docker compose up –build -d
docker compose logs -f backend
docker compose down -v
“`

### Step 4 — Verify the Dashboard

Walk through these checks in the browser after `./build.sh demo`:

1. Command Center KPI cards show non-zero numbers (not dashes or zeros)
2. Charts render with visible data points
3. Drag a widget, refresh the page — layout stays where you left it
4. performance page shows WebSocket as Connected with live points updating
5. Template Gallery lists templates and Deploy Template creates a new board

**Success criteria:** `metric_samples > 0`, chart `points > 0`, E2E checks all pass, dashboard KPIs update after demo injection.

### Stop and Clean Up

When you are done for the day:

“`bash
./stop.sh # stop host processes and Docker containers
./cleanup.sh # remove node_modules, venv, caches, prune Docker resources
“`

—

## Key Source Files

If you get lost in the codebase, start with these files.

—

## Troubleshooting

—

## Assignment

Build a **custom widget type** `alert_summary` that displays the top 5 services by error-rate from the aggregated API. Add it to the widget library, persist it on a dashboard, and verify layout survives page refresh.

### Solution Hints

1. Extend `WidgetRenderer` with a new case calling `/API/v1/charts/aggregated?group_by=service`
2. Register the type in `WidgetLibrary` with an appropriate icon
3. POST to `/API/v1/dashboards/{id}/widgets` with `widget_type: alert_summary`
4. Confirm layout JSON in GET `/API/v1/dashboards/{id}` includes your widget’s grid position

—

*Next: User & Team Management — multi-tenant dashboards with RBAC inheritance.*

- Hands-On Tutorial

system11

Hands-On System Design Tutorial

## Lesson 9: Data Export & Reporting Platform
## What We Are Building

InfraWatch **Reports** — a production-grade export and reporting control plane:

1. **Export Engine** — Multi-format streaming exports (CSV, JSON, Excel, PDF) with checksum validation
2. **Report Templates** — Jinja2-driven dynamic reports with preview and scheduled delivery
3. **Visualization API** — Chart endpoints with pre-aggregated hourly metrics and Redis caching
4. **Advanced Analytics** — Statistical baselines, anomaly detection, forecasting, and correlation matrices
5. **performance Layer** — Query caching, index recommendations, Prometheus metrics, slow-query tracking
6. **Operator Console** — Grafana-inspired dashboard with export wizard, real-time progress, and history

—

## Core Concept

Export systems in production (Datadog, Grafana, Splunk) never dump raw tables to users. They **stream, compress, and cache** query results because a single unbounded export can saturate database I/O and starve live dashboards. The non-obvious insight: treat exports as **asynchronous jobs with TTL-bound artifacts**, not synchronous API responses. Grafana’s report renderer and Datadog’s scheduled snapshots both follow this pattern — queue the work, stream batches, validate integrity, expire the file.

—

## Where This Fits

“`
InfraWatch Realtime (live streams)
│
InfraWatch Reports ◀ this module
│
InfraWatch Dashboard (widgets)
“`

Reports consumes aggregated metrics and notification data upstream, producing downloadable artifacts and scheduled executive summaries downstream.

—

## Component Architecture

The API sits in the middle. The React console talks to it over REST and WebSocket. PostgreSQL stores your metrics and export jobs. Redis caches chart queries so repeated dashboard loads do not hammer the database.

—

## Inside the Project

Once you open the repository, the layout maps directly to those services:

“`
infrawatch-reports/
├── backend/
│ ├── app/
│ │ ├── main.py # FastAPI app, lifespan, WebSocket
│ │ ├── API/ # exports, templates, schedules, analytics
│ │ ├── core/ # config, database
│ │ ├── models/ # ExportJob, MetricSample, etc.
│ │ ├── schemas/ # Pydantic request/response
│ │ └── services/ # export_engine, analytics, cache, seed
│ ├── tests/test_api.py
│ └── requirements.txt
├── frontend/
│ ├── src/pages/ # 7 console pages
│ ├── src/API/client.ts
│ └── package.json
├── docker-compose.yml
├── build.sh
├── stop.sh
└── cleanup.sh
“`

You do not need to memorize every file. Focus on three folders: `API/` (routes), `services/` (business logic), and `frontend/src/pages/` (what the operator sees).

—

## How the Pieces Work Together

**Export engine.** A POST to `/API/v1/exports` creates a job. The job moves through `pending`, `processing`, and `completed` (or `failed` / `expired`). Rows stream in batches, the file gets an MD5 checksum, and downloads expire after 24 hours.

**Report templates.** Jinja2 templates support variable extraction, live preview, and scheduled generation. A weekly summary template is seeded on first startup.

**Analytics pipeline.** Raw metric samples roll up into hourly buckets. Chart endpoints check Redis first (60-second TTL), then PostgreSQL. Anomaly detection uses z-scores; forecasting uses linear regression; correlations use Pearson coefficients.

**performance layer.** In-memory cache plus Redis, latency histograms at P50/P95/P99, connection pool stats, and a Prometheus endpoint at `/API/v1/performance/prometheus`.

—

## Control & Data Flow

1. Operator selects data source and format in the export wizard
2. API checks Redis cache for chart queries; misses hit PostgreSQL with indexed scans
3. Export engine streams rows in batches, writes file, computes MD5 checksum
4. WebSocket pushes progress stages: fetching → converting → complete
5. Completed files expire after 24 hours; Prometheus records latency percentiles

—

## Export Job Lifecycle

Every export job follows this path. If something breaks during processing, the job lands in `failed` with an error message. If nobody downloads within 24 hours, it moves to `expired`.

—

## API Reference

### Exports

| Endpoint | Method | Purpose |
|———-|——–|———|
| `/API/v1/exports` | POST | Create export job |
| `/API/v1/exports` | GET | List jobs |
| `/API/v1/exports/{id}` | GET | Job status |
| `/API/v1/exports/{id}/download` | GET | Download file |
| `/API/v1/exports/{id}` | DELETE | Cancel job |

### Templates and Schedules

| Endpoint | Method | Purpose |
|———-|——–|———|
| `/API/v1/templates` | GET/POST | Template CRUD |
| `/API/v1/templates/{id}/preview` | POST | Render preview |
| `/API/v1/schedules` | GET/POST | Cron schedules |
| `/API/v1/schedules/run-due` | POST | Trigger due schedules |

### Analytics

| Endpoint | Purpose |
|———-|———|
| `GET /API/v1/analytics/chart` | Multi-series chart data |
| `GET /API/v1/analytics/trends` | Direction, percent change, moving average |
| `GET /API/v1/analytics/compare` | Dimension comparison |
| `GET /API/v1/analytics/drilldown` | Channel to template hierarchy |
| `GET /API/v1/analytics/statistics/{metric}` | mean, p95, p99 |
| `GET /API/v1/analytics/anomalies` | z-score detection |
| `GET /API/v1/analytics/predictions/{metric}` | Forecast with confidence band |
| `GET /API/v1/analytics/correlations/{metric}` | Cross-metric Pearson r |

### Operations

Quick API check after the stack is running:

“`bash
curl -s http://localhost:5050/API/health
curl -s http://localhost:5050/API/v1/operations/overview | python3 -m json.tool
“`

—

## Operator Console

The UI uses a Grafana-inspired dark theme. Each sidebar page maps to one part of the platform:

Walk every page after startup. If KPIs show zero, run the demo step below before assuming something is broken.

—

## Build, Test and Demo

### Prerequisites

Python 3.12+, Node.js 22+, and Docker with Compose v2. Ports 5050, 5220, 5480, and 6430 must be free.

### Step 1 — Enter the project and make scripts executable

“`bash
cd week9/infrawatch-reports
chmod +x build.sh stop.sh cleanup.sh
“`

### Step 2 — Run tests

“`bash
./build.sh test
“`

This runs 14 pytest cases (health, charts, exports, download, end-to-end) and 1 vitest case (UI renders). All should pass before you start the stack.

### Step 3 — Start the stack (hybrid mode, recommended)

Postgres and Redis run in Docker; API and UI run on the host.

“`bash
./build.sh local
“`

Open **http://localhost:5220**. The API listens on **http://localhost:5050**.

### Step 4 — Run the demo validation

With the stack running:

“`bash
./build.sh demo
“`

**Expected demo output:**

“`
{“status”:”healthy”,”service”:”infrawatch-reports”}
Metric samples: 2500+ | Delivery volume: >0
{“passed”: true, “chart_points”: >0, “export_rows”: >0}
Open http://localhost:5220
“`

### Step 5 — Walk the UI manually

1. **Overview** — confirm KPI cards are not zero; click **Generate Sample Export**
2. **Exports** — click Next twice, then **Start Export**; wait for the spinner; download the file
3. **Analytics** — open each tab; charts should render with real data
4. **performance** — CPU and latency values should update

### Full pipeline in one command

“`bash
./build.sh all
“`

Runs test, local start, and demo in sequence.

### With Docker (full stack)

“`bash
./build.sh docker
“`

Requires Docker Hub access for `python:3.12-slim`, `node:22-alpine`, and `nginx:alpine`. If Hub is unreachable, the script falls back to hybrid mode automatically.

Pre-pull images when your network is stable:

“`bash
docker pull python:3.12-slim node:22-alpine nginx:alpine postgres:16-alpine redis:7-alpine
./build.sh docker
“`

On WSL, if you see `lookup registry-1.docker.io: no such host`:

“`bash
echo “nameserver 8.8.8.8” | sudo tee /etc/resolv.conf
“`

Restart Docker Desktop, then retry.

### Stop and clean up

“`bash
./stop.sh # stop host processes and compose stack
./cleanup.sh # remove node_modules, venv, caches, project Docker images
“`

—

## Success Criteria

– [ ] `./build.sh test` — all backend and frontend tests pass
– [ ] Overview KPIs show non-zero metric samples and delivery volume
– [ ] Export wizard creates downloadable CSV/JSON/Excel/PDF files
– [ ] Analytics tabs render comparison charts, trends, and correlations
– [ ] performance page shows live CPU and query latency percentiles
– [ ] Templates and schedules pages display seeded data

—

## Troubleshooting

—

## Key Files to Know

—

## Assignment

Add a custom report template that includes a bar chart of failure rates per channel. Schedule it to run every Monday at 08:00 UTC and deliver via email recipients list.

**Hints:** Create a Jinja2 template with a `{% for %}` loop over channel data. Use the `/API/v1/analytics/compare` endpoint variables in the template context. Register a cron schedule via the Schedules API with expression `0 8 * * 1`.

—

## Solution Hints

1. Fetch comparison data: `GET /API/v1/analytics/compare?metric_name=failure_rate`
2. POST to `/API/v1/templates` with HTML containing `{% for item in channels %}`
3. POST to `/API/v1/schedules` with `cron_expression: “0 8 * * 1″` and `email_recipients: [“you@domain.com”]`
4. Trigger manually: `POST /API/v1/schedules/run-due` after setting `next_run_at` to past

—

*Next: Advanced Dashboard Features — drag-and-drop widgets consuming these chart APIs.*

- Hands-On Tutorial

system11

Hands-On System Design Tutorial

## Lesson 8: Real-time Streaming Control Plane

If you have ever watched a Datadog dashboard update live, or seen PagerDuty fire an alert the second a server crosses a threshold, you have already seen real-time infrastructure monitoring in action. Today you will build that transport layer yourself — not as a toy chat app, but as a production-grade streaming control plane called **InfraWatch Realtime**.

By the end of this lesson you will have a working operator console with live charts, authenticated WebSocket connections, throttled metric delivery, and a reliability lab that measures p99 latency under load. Let us get into it.

—

## What We Are Building

InfraWatch **Realtime** — a production streaming control plane for infrastructure operators:

1. **Connection Hub** — WebSocket transport with JWT auth, rooms, and reconnection tracking
2. **Data Streams** — Live metric streaming, alert broadcasting, and throttled delivery
3. **Event Protocol** — Versioned envelopes, client sync, conflict resolution, offline queue
4. **performance Layer** — Redis queuing, bandwidth optimization, circuit breakers
5. **Reliability Lab** — Load tests, concurrency bursts, and p99 benchmarking
6. **Live Console** — Real-time charts, connection status, auto-refresh, offline fallback
7. **Operations** — End-to-end orchestration with graceful degradation and recovery

—

## Core Concept

Real-time infrastructure monitoring is not about pushing raw data faster — it is about **controlled delivery under load**. Datadog Live Processes, PagerDuty incident streams, and Grafana live annotations all share the same pattern: authenticate connections, subscribe to topics, throttle noisy metrics, and degrade gracefully when backends saturate.

The non-obvious insight: **the WebSocket is a control channel, not a firehose**. Production systems batch, compress, and prioritize — critical alerts bypass throttles. When a client disconnects, events queue locally and flush on reconnect with version-based conflict resolution, the same last-write-wins pattern Stripe uses for dashboard state sync.

—

## Where This Fits

“`
InfraWatch Alerts + Notify
│
InfraWatch Realtime ◀ this module
│
InfraWatch Export / Dashboard
“`

Realtime sits downstream of alerts and notifications, providing the live transport layer that powers operator consoles across the platform.

—

## Component Architecture

Think of the backend as air traffic control. It does not fly the planes (collect metrics on every server in your fleet) — it routes, prioritizes, and delivers updates to the right screens at the right speed.

—

## Control & Data Flow

1. Operator UI opens WebSocket to `/ws/{client_id}`
2. Client subscribes to `metrics_update`, `alerts_critical`, `alerts_warning`
3. Background simulator collects psutil metrics every 500ms
4. StreamManager throttles and batches; critical alerts bypass the queue
5. Events broadcast to subscribed clients; offline clients queue for reconnect flush
6. Dashboard polls REST fallback when WebSocket is down

—

## Connection Lifecycle

A connection is never just “on” or “off” in production. It moves through distinct states — connecting, streaming, offline, syncing queued events, and degraded when backends are under stress. Your UI should reflect each state so operators know whether they are looking at live data or a cached fallback.

—

## Building Each Layer

The sections below map directly to the seven capabilities listed above. Work through them in order — each layer depends on the one before it.

### 1. Connection Hub

The connection manager (`backend/app/services/connection_manager.py`) is the front door. Every client that wants live data must pass through it.

What it does:
– Issues and verifies JWT tokens on WebSocket connect
– Keeps a registry of every active client and which room they belong to
– Handles `join_room` / `leave_room` with broadcast notifications to other members
– Tracks reconnection counts and messages-per-second throughput

**REST endpoints**

**WebSocket protocol** — connect to `WS /ws/{client_id}?token=…`

| Client sends | Server replies |
|————–|—————-|
| `ping` | `pong` |
| `subscribe` with `{topics:[]}` | `subscribed` |
| `join_room` | `room_joined` |

Check it works:

“`bash
curl -s http://localhost:5050/API/v1/connections/stats | python3 -m json.tool
“`

You should see `active_connections` and `messages_per_second` fields in the response.

—

### 2. Data Streams

Once clients are connected, they need something to stream. The StreamManager handles topic subscriptions and delivery pacing.

What it does:
– Lets each client subscribe to specific topics (`metrics_update`, `alerts_critical`, etc.)
– Batches outgoing messages on a 100ms flush interval so you do not flood the browser
– Lets critical alerts jump the queue — warnings can wait, outages cannot
– Tracks gzip compression savings for bandwidth reporting

The metric collector uses `psutil` to read real CPU, memory, and disk numbers from the host machine and pushes them into a rolling history buffer. No mock data — the numbers on your dashboard are live readings from your own machine.

**REST endpoints**

Check it works:

“`bash
curl -s http://localhost:5050/API/v1/streaming/metrics/current
“`

Expected output looks like: `{“cpu_percent”: 24.5, “memory_percent”: 58.2, …}` — values will differ on your machine, but they should never be zero after the stack is running.

—

### 3. Event Protocol

Raw WebSocket messages become hard to manage at scale. The event protocol wraps every message in a standard envelope: `version`, `type`, `id`, `timestamp`, `payload`, `metadata`.

What it does:
– **SyncEngine** — detects version conflicts and resolves them with last-write-wins
– **OfflineQueue** — stores pending events per client; drains them automatically on reconnect

Check it works:

“`bash
curl -s -X POST http://localhost:5050/API/v1/events/publish
-H ‘Content-Type: application/json’
-d ‘{“type”:”STATUS”,”payload”:{“service”:”API”},”client_id”:”test”}’
“`

—

### 4. performance Layer

When hundreds of operators connect at once, you need backpressure. The performance layer prevents the hub from collapsing under its own output.

What it does:
– **RedisQueue** — priority queues (`critical`, `normal`, `low`) with in-memory fallback during tests
– **CircuitBreaker** — per-subsystem breakers (`streaming`, `alerts`, `events`) that trip open when error rates spike
– **PerformanceMonitor** — aggregates queue depth, memory usage, pool size, and delivery counters

Check it works:

“`bash
curl -s http://localhost:5050/API/v1/performance/snapshot | python3 -m json.tool
“`

—

### 5. Reliability Lab

You cannot trust a real-time system until you have measured it under load. The load test harness simulates N simultaneous WebSocket connections, records p99 latency, and stores results in the `load_test_runs` table.

Check it works:

“`bash
curl -s -X POST http://localhost:5050/API/v1/testing/start
-H ‘Content-Type: application/json’
-d ‘{“name”:”guide_test”,”connections”:25}’
“`

Expected: `”status”: “completed”` and `”success_rate”: 100.0`.

—

### 6. Live Console UI

The React frontend uses a `RealtimeContext` provider that owns the WebSocket lifecycle for the entire app:

– Exponential backoff on reconnect (waits longer after each failed attempt)
– Offline message queue that flushes automatically when the socket comes back
– Auto-refresh toggle so operators can pause live updates without disconnecting

**Console pages**

—

### 7. Operations Orchestration

The RealtimeHub runs a background simulator that pushes metrics through the full throttle pipeline. Operations endpoints let you test the whole system end-to-end:

—

## Key Files

—

## Getting the Project

The runnable project lives in `infrawatch-realtime/`. If you need to regenerate it from scratch (for example after cloning only the lesson files), run the backup generator from the parent directory:

“`bash
./setup.sh generate
“`

That recreates the full `infrawatch-realtime/` directory from embedded source files in the script. Verify the output with:

“`bash
./setup.sh verify
“`

—

## Prerequisites

– Python 3.12 or newer
– Node.js 22 or newer
– Docker with Compose v2

—

## Build, Test and Demo

All commands below run from inside `infrawatch-realtime/`:

“`bash
cd infrawatch-realtime
chmod +x build.sh stop.sh
“`

### Run tests only

“`bash
./build.sh test
“`

This runs 9 backend pytest cases and 1 frontend vitest case, then builds the production frontend bundle. All tests must pass before you start the stack.

### Start local development (without full Docker)

Postgres and Redis run in Docker containers. The API and UI run directly on your machine — faster iteration, easier debugging.

“`bash
./build.sh local
“`

When healthy, you will see:

“`
Backend healthy on :5050
Frontend on http://localhost:5220
“`

### Start the full Docker stack

Everything — backend, frontend, Postgres, Redis — runs inside containers. Use this when you want a clean, reproducible environment.

“`bash
./build.sh docker
“`

### Run the functional demo

The stack must already be running (`local` or `docker`):

“`bash
./build.sh demo
“`

**Expected demo output:**

“`
{“status”:”healthy”,”service”:”infrawatch-realtime”}
{“simulated”:true,”alerts_created”:2}
Alerts: 5 | Events: 2
{“passed”: true, …}
Open http://localhost:5220
“`

### Run everything in one shot

“`bash
./build.sh all
“`

Runs tests, starts the local stack, and executes the demo automatically.

### Stop all services

“`bash
./stop.sh
“`

—

## Walking the Dashboard

Open http://localhost:5220 and click through every sidebar section. KPIs should show non-zero values after you run the demo or click **Run Live Burst** on the Command Center.

What to look for on each page:

– **Command Center** — live CPU chart updating, 8 KPI cards with real numbers
– **Connections** — session table showing active WebSocket clients
– **Data Streams** — metric history and alert feed with throttle statistics
– **Event Protocol** — sync state panel and conflict counter
– **performance** — Redis queue depth, memory usage, circuit breaker status
– **Reliability Lab** — run a load test and confirm p99 latency appears
– **Live Console** — connection indicator (green = connected) and offline queue size
– **Operations** — trigger degradation, then recovery, and watch the status change

—

## Docker & Runtime

### Without Docker (local dev)

Postgres and Redis run in Docker; API and UI run on the host.

“`bash
cd infrawatch-realtime
chmod +x build.sh stop.sh
./build.sh local
“`

### With Docker (reproducible)

“`bash
./build.sh docker
“`

### Stop

“`bash
./stop.sh
“`

—

## Success Criteria

– [ ] `./build.sh test` — 9 pytest + 1 vitest pass
– [ ] `./build.sh local` — API on :5050, UI on :5220
– [ ] `./build.sh demo` — alerts > 0, events > 0, E2E passed
– [ ] Command Center shows live CPU chart and 8 KPI cards
– [ ] Connections page shows auth, rooms, session table
– [ ] Data Streams shows metrics, alert feed, throttle stats
– [ ] Event Protocol shows sync state and conflict counter
– [ ] performance shows queue depth, memory, circuit breakers
– [ ] Reliability Lab runs load test with p99 results
– [ ] Live Console shows connection status and offline queue
– [ ] Operations triggers degradation and recovery

—

## Troubleshooting

—

## Assignment

1. Add a new subscription topic `deployments` that broadcasts deployment status changes
2. Implement per-room message history (last 50 messages) with a UI panel on the Connections page
3. Add a chaos test that simulates 200 rapid reconnects and reports success rate in Reliability Lab

## Assignment Hints

– **Task 1:** Extend `StreamManager.subscribe()` topics in `backend/app/services/stream_manager.py`; add a broadcast route in `streaming.py`; subscribe from `RealtimeContext.tsx`
– **Task 2:** Store room messages in a new `RoomMessage` model; expose `GET /API/v1/connections/rooms/{room}/history`
– **Task 3:** Add `POST /API/v1/testing/chaos/reconnect` in `testing.py`; mirror the load harness pattern from `LoadTestHarness`

- Hands-On Tutorial

system11

Hands-On System Design Tutorial

## Lesson 7 : Notification Dispatch Control Plane

## What We Are Building

InfraWatch **Notify** — a production notification dispatch console modeled after PagerDuty and Opsgenie:

1. **Multi-channel delivery** — Email, SMS, Slack, webhook, and push providers behind a unified interface
2. **Template engine** — Jinja2 rendering with per-channel formats and preview
3. **Delivery pipeline** — Redis priority queue, background worker, retry with backoff, rate limiting
4. **Preference routing** — Severity-based channel selection, quiet hours, suppression rules
5. **Reliability layer** — Circuit breakers, delivery tracking timeline, integration test harness
6. **Operator console** — Dark-theme dashboard with live WebSocket feed and channel health

—

## Core Concept: Alerts Are Inputs, Notifications Are Outputs

PagerDuty does not blast every engineer for every metric spike. It routes **alert events** through **user preferences**, **escalation policies**, and **channel capacity** before a human ever sees a ping. The non-obvious insight: **notification delivery is its own distributed system** — separate from alert detection, separate from metric storage.

A firing alert is a signal. A notification is a **delivery attempt** with its own lifecycle: queued → processing → delivered or failed → retry. Datadog and Grafana solve the “what broke” problem; Notify solves “who gets told, how, and did it actually arrive.”

—

## Where This Fits

“`
Alert Management decides something needs attention
│
Notification Dispatch ◀ this lesson
│
Human Response on-call engineer acts
“`

Notify sits downstream of alert systems (like InfraWatch Alerts). It assumes alerts arrive with severity, service context, and a message — then handles the hard part of reliable multi-channel fan-out.

—

## Component Architecture

**Orchestrator** — Receives alerts, resolves recipients, renders templates, enqueues delivery jobs.
**Preference Router** — Applies quiet hours, severity-channel matrix, and global opt-out.
**Queue Manager** — Redis-backed priority heap; CRITICAL jobs dequeue before LOW.
**Delivery Worker** — Background thread processes batches, records events, handles retries.
**Channel Providers** — Pluggable senders with validation and latency tracking.
**WebSocket Hub** — Pushes delivery state changes to the operator console.

—

## Control & Data Flow

Alert POST → preference check per user → template render per channel → idempotency key → enqueue → worker dequeue → circuit breaker check → channel send → delivery event logged → WebSocket broadcast → dashboard card refresh.

Escalation worker polls unacknowledged alerts. After the configured timeout, it bumps escalation level and re-triggers fan-out to the next tier — exactly how Opsgenie tiered paging works.

When you trace a single alert through the code, this is the path it follows:

1. Operator or API POSTs alert to `/API/v1/alerts/`.
2. Orchestrator loads all user preferences.
3. `PreferenceService.should_send()` checks global enable, quiet hours, and severity exceptions.
4. Channels resolved per severity (user override or defaults).
5. Jinja2 template rendered per channel (`alert_email`, `alert_sms`, etc.).
6. Idempotency key prevents duplicate delivery for same alert + user + channel.
7. Notification enqueued in Redis priority queue.
8. Delivery worker dequeues, checks circuit breaker and rate limit, sends via channel provider.
9. Delivery events logged; WebSocket broadcasts status to dashboard.

—

## Notification State Machine

Each notification record tracks its own state independently of the parent alert.

The parent alert moves through its own states separately:

—

## Design Notes Worth Remembering

These patterns show up in real on-call systems. You implemented smaller versions of each one:

– **Alerts are inputs, notifications are outputs** — Each delivery attempt has its own state machine, independent of the parent alert. PagerDuty and Opsgenie work the same way.
– **Idempotency keys** — SHA-256 of `alert_id + user_id + channel` stops duplicate pages during retries.
– **Priority queue** — CRITICAL jobs dequeue before LOW via Redis sorted-set scores.
– **Circuit breaker** — After three consecutive channel failures, Redis key `notify:circuit:{CHANNEL}` blocks attempts for 30 seconds.
– **Quiet hours** — Timezone-aware suppression with severity exceptions (CRITICAL always delivers).
– **Webhook receiver** — Built-in `POST /API/v1/channels/webhook-receiver` lets you test webhooks locally without external services.
– **No API keys required** — Channel providers validate and dispatch in development. Set `SECRET_KEY` via `.env` in production. database credentials are local Docker defaults only.

—

## Prerequisites

Before you start, confirm you have:

– Python 3.12 or newer
– Node.js 22 or newer
– Docker with Compose v2

—

## Build, Test, and Run

All commands run from inside `infrawatch-notify/`.

### Step 1 — Install dependencies

“`bash
cd infrawatch-notify
chmod +x build.sh stop.sh cleanup.sh

python3 -m venv backend/.venv
source backend/.venv/bin/activate
pip install -r requirements-dev.txt

cd frontend && npm install && cd ..
“`

### Step 2 — Run tests

“`bash
./build.sh test
“`

You should see 8 backend tests pass, one frontend smoke test pass, and a successful Vite production build.

To run tests individually:

“`bash
# Backend
cd backend
export PYTHONPATH=.
export TESTING=1
export DATABASE_URL=”sqlite://”
pytest -q –tb=short

# Frontend
cd frontend
npm run test — –run
npm run build
“`

### Step 3 — Start the stack (local mode)

“`bash
./build.sh local
“`

Docker starts Postgres and Redis. The API and UI run on your machine.

Expected output:

“`
[build] Postgres ready
database seeded
[build] Backend healthy on :5030
[build] Frontend on http://localhost:5200
“`

Open http://localhost:5200 in your browser.

### Step 4 — Run the demo

With the stack running:

“`bash
./build.sh demo
“`

Or do everything in one pass:

“`bash
./build.sh all
“`

The demo fires sample alerts, waits for the delivery worker, and prints notification stats. Delivered count should be greater than zero.

### Step 5 — Stop and clean up

“`bash
./stop.sh # stop processes and containers
./cleanup.sh # remove artifacts and prune Docker
“`

—

## Docker

### Hybrid mode (what `./build.sh local` uses)

Only database services run in Docker:

“`bash
docker compose up -d postgres redis
“`

The app connects via `localhost:5460` (Postgres) and `localhost:6410` (Redis).

### Full stack in containers

“`bash
./build.sh docker
“`

Frontend nginx proxies `/API` and `/ws` to the backend container.

Tear down:

“`bash
docker compose down -v –remove-orphans
“`

—

## Working with the API

Base URL: `http://localhost:5030`

### Health check

“`bash
curl -sS http://localhost:5030/API/health | python3 -m json.tool
“`

Expected:

“`json
{“status”: “healthy”, “service”: “infrawatch-notify”}
“`

### Create an alert (triggers fan-out)

“`bash
curl -sS -X POST http://localhost:5030/API/v1/alerts/
-H “Content-Type: application/json”
-d ‘{
“service_name”: “API-gateway”,
“severity”: “CRITICAL”,
“title”: “CPU saturation”,
“message”: “CPU at 96% on prod-us-east-1”
}’
“`

### Simulate a dispatch burst

“`bash
curl -sS -X POST http://localhost:5030/API/v1/alerts/simulate | python3 -m json.tool
“`

### Check notification stats

“`bash
curl -sS http://localhost:5030/API/v1/notifications/stats | python3 -m json.tool
“`

### Test all five channels

“`bash
curl -sS -X POST http://localhost:5030/API/v1/testing/integration | python3 -m json.tool
“`

### Endpoint reference

| Method | Path | Purpose |
|——–|——|———|
| GET | `/API/health` | Health check |
| POST/GET | `/API/v1/alerts/` | Create and list alerts |
| POST | `/API/v1/alerts/simulate` | Demo alert burst |
| POST | `/API/v1/alerts/{id}/acknowledge` | Acknowledge alert |
| POST | `/API/v1/alerts/{id}/resolve` | Resolve alert |
| GET | `/API/v1/notifications/` | List notifications |
| GET | `/API/v1/notifications/stats` | Dashboard aggregates |
| GET | `/API/v1/notifications/tracking/{id}` | Delivery event timeline |
| GET/PUT | `/API/v1/preferences/{user_id}` | User delivery preferences |
| GET | `/API/v1/templates/` | List message templates |
| POST | `/API/v1/templates/render` | Preview template |
| GET | `/API/v1/channels/status` | Per-channel health |
| POST | `/API/v1/channels/test` | Send test notification |
| POST | `/API/v1/testing/integration` | All-channel integration test |
| WS | `/ws/notifications` | Live delivery updates |

—

## Dashboard Walkthrough

Open http://localhost:5200 after `./build.sh local`.

Metrics stay at zero until alerts are dispatched. Click **Run Dispatch Burst** or run the simulate curl command, wait 3–5 seconds for the delivery worker, then refresh.

—

## Success Criteria

After running `./build.sh all`:

– Dashboard shows **non-zero** total and delivered notification counts
– **Run Dispatch Burst** creates alerts and fans out across all five channels
– Channel health panel shows success/failure counters updating
– Preferences page toggles quiet hours live
– Integration test reports **5/5 channels passed**

—

## Troubleshooting

—

## Assignment

Add a **digest mode** preference: when enabled, non-CRITICAL notifications batch into a single email every 15 minutes instead of immediate delivery.

**Steps:**
1. Add `digest_enabled` and `digest_interval_minutes` fields to `UserPreference`
2. Modify the orchestrator to queue digest-eligible notifications separately
3. Create a periodic task that renders a combined Jinja2 template and sends one email
4. Add a toggle on the Preferences UI page
5. Write a test proving three MEDIUM alerts produce one digest email

—

## Assignment Solution Hints

– Store digest candidates in a Redis sorted set keyed by `user_id`, scored by timestamp
– The digest worker should use `ZPOPMIN` to grab expired batches
– Reuse `alert_email` template with a `{% for alert in alerts %}` loop
– CRITICAL severity must bypass digest regardless of preference — check severity before enqueue
– Test: create 3 alerts with `severity=MEDIUM`, wait for digest interval, assert `Notification` count for EMAIL equals 1

—

## Practical Takeaway

You built the layer that turns infrastructure signals into human action. The queue, retry logic, and circuit breaker are the same patterns Twilio, SendGrid, and PagerDuty use at scale — just smaller. Next time an on-call page fails silently, you will know exactly which state transition broke.

- Hands-On Tutorial

system11

Hands-On System Design Tutorial

## Lesson 6 : Alert Management Control Plane

## What We Are Building

InfraWatch **Alerts** — a production alert management console modeled after PagerDuty and Datadog:

1. **Rule engine** — Threshold, anomaly, and expression-based rules with templates
2. **Evaluation pipeline** — Sustained breach detection, deduplication, suppression windows
3. **Lifecycle processing** — Severity scoring, escalation policies, auto-resolution
4. **Query plane** — Search, statistics, bulk operations, CSV/JSON export
5. **Operator console** — Dark-theme dashboard with live WebSocket updates
6. **Notification routing** — Channel delivery with circuit-breaker protection

—

## Core Concept: Signal vs Noise

Every monitoring platform ingests thousands of metric points per minute. The hard problem is not detection — it is **deciding which signals deserve human attention**. Datadog and PagerDuty solve this with pending timers (avoid flapping), fingerprint deduplication (one alert per root cause), and suppression windows (maintenance silence).

The non-obvious insight: **alert state is a first-class entity**, not a side effect of rule evaluation. A rule fires; an alert instance is born with its own lifecycle. Operators acknowledge, escalate, and resolve *alerts* — not rules.

—

## Where This Fits

“`
Metrics Collection raw telemetry
│
Background Processing scheduled evaluation jobs
│
Alert Management ◀ this lesson
│
Notifications delivers outcomes to humans
“`

Alerts sit between completed metric pipelines and human response. They assume metrics arrive reliably and workers can evaluate rules on schedule.

—

## Component Architecture

**Ingest** — Agents POST metric samples; each sample triggers rule evaluation.
**Evaluator** — Compares values against thresholds or Z-score anomaly models.
**Processor** — Manages pending→firing transitions, severity scoring, history audit.
**Query API** — Powers search, statistics, and export for the operator console.
**WebSocket** — Pushes state changes to the dashboard without polling.

—

## Control & Data Flow

Metric arrives → stored in PostgreSQL → evaluator checks all enabled rules → suppression filter → dedup fingerprint → create or update alert instance → pending timer → firing → notification router → WebSocket broadcast → dashboard card refresh.

Step-by-step path your code follows:

1. Agent or simulator POSTs a sample to `/API/v1/metrics/ingest`.
2. Sample lands in the `metric_samples` table.
3. The evaluation engine loads every enabled rule.
4. The suppression service checks maintenance windows and regex patterns.
5. The evaluator compares the value against a threshold or Z-score anomaly model.
6. An alert instance is created or updated — pending becomes firing after `for_duration_seconds`.
7. The notification router logs delivery; WebSocket pushes the update.
8. Dashboard stats cards and the alert table refresh.

—

## Alert Lifecycle

States: **OK → Pending → Firing → Acknowledged → Resolved**, with a **Suppressed** branch during maintenance windows. Pending requires sustained breach (`for_duration_seconds`) before firing — the same pattern Prometheus `for:` clauses use.

—

## Real-World Context

– **PagerDuty** routes firing alerts through escalation policies with time-delayed notifications.
– **Datadog** monitors use evaluation windows to prevent alert storms from brief spikes.
– **Grafana** alert rules separate rule definition from alert instance management.

Your console implements all three patterns in one self-contained stack.

—

## Design Notes

These choices show up directly in the code you will run:

– **Pending before firing** — Rules use `for_duration_seconds` so a one-second CPU spike does not wake anyone at 3 a.m.
– **Fingerprint dedup** — A SHA-256 fingerprint per rule and metric stops duplicate alert rows from flooding the table.
– **DB-first state** — Every transition is written to `alert_state_history` so you can audit what happened and when.
– **Same-origin WebSocket** — The frontend connects to `ws:///ws/alerts`; Vite and nginx proxy it correctly in both local and Docker modes.
– **Self-contained project** — Clone `infrawatch-alerts/` alone. Everything you need is inside that folder.
– **No API keys in source** — database credentials are local Docker defaults for development. Set `SECRET_KEY` in `.env` before production.

—

## Project Layout

“`
infrawatch-alerts/
├── backend/
│ ├── app/
│ │ ├── main.py # FastAPI routes + WebSocket + evaluation loop
│ │ ├── core/config.py # pydantic-settings
│ │ ├── core/database.py # SQLAlchemy engine + session
│ │ ├── models/alert.py # AlertRule, AlertInstance, EscalationPolicy, etc.
│ │ ├── schemas/alert.py # Pydantic request/response models
│ │ ├── services/
│ │ │ ├── evaluator.py # Threshold + anomaly evaluation, dedup
│ │ │ ├── processor.py # Lifecycle, severity, notifications
│ │ │ ├── query.py # Search, statistics, export
│ │ │ ├── suppression.py # Maintenance window logic
│ │ │ ├── validator.py # Rule validation + test harness
│ │ │ └── seed.py # Default rules and templates
│ │ ├── API/
│ │ │ ├── rules.py # Rule CRUD, templates, testing
│ │ │ ├── alerts.py # Search, ack, resolve, bulk ops
│ │ │ └── metrics.py # Ingest + simulate
│ │ └── websocket/manager.py # Live dashboard push
│ ├── tests/
│ ├── requirements.txt
│ ├── requirements-dev.txt
│ └── Dockerfile
├── frontend/
│ ├── src/
│ │ ├── pages/Dashboard.tsx # Command center + metric simulator
│ │ ├── pages/Alerts.tsx # Searchable alert explorer
│ │ ├── pages/Rules.tsx # Rule list + test dialog
│ │ ├── pages/Templates.tsx # Template gallery
│ │ ├── components/AlertTable.tsx # Bulk ack/resolve
│ │ ├── components/MetricSimulator.tsx
│ │ ├── API/client.ts
│ │ └── hooks/useWebSocket.ts
│ ├── package.json
│ ├── vite.config.ts
│ └── Dockerfile
├── docker-compose.yml
├── build.sh # test | local | docker | demo | all
├── start.sh
├── stop.sh
├── cleanup.sh
├── requirements.txt
├── requirements-dev.txt
├── .env.example
└── README.md
“`

—

## Success Criteria

After completing this lesson you should be able to:

– [ ] Create threshold and anomaly rules via API
– [ ] Ingest metrics and observe alerts transition through pending → firing
– [ ] Acknowledge and bulk-resolve alerts from the dashboard
– [ ] Search, export, and view severity statistics
– [ ] See live updates via WebSocket without page refresh

—

## Build, Test, and Run

Everything below runs from inside the `infrawatch-alerts/` directory. You need Python 3.12+, Node.js 22+, and Docker with Compose v2.

### First-time setup

“`bash
cd infrawatch-alerts
chmod +x build.sh stop.sh start.sh cleanup.sh

python3 -m venv backend/.venv
source backend/.venv/bin/activate
pip install -r requirements-dev.txt

cd frontend && npm install && cd ..
“`

Copy environment defaults when you need them:

“`bash
cp .env.example .env
“`

### Run tests

“`bash
./build.sh test
“`

You should see 11 pytest tests pass, a Vitest smoke test pass, and a successful Vite production build.

To run test suites individually:

“`bash
cd backend
export PYTHONPATH=.
export TESTING=1
export DATABASE_URL=”sqlite://”
pytest -q –tb=short

cd ../frontend
npm run test — –run
npm run build
“`

### Start the stack (local mode)

“`bash
./build.sh local
“`

Docker starts Postgres and Redis only. The API and UI run on your machine. Wait for:

“`
[build] Postgres ready
database seeded
[build] Backend healthy on :5020
[build] Frontend on http://localhost:5190
“`

Open http://localhost:5190

### Generate metrics and see alerts

Metrics do not appear until data is ingested. Pick either path:

**From the dashboard** — Click **Run Production Burst**, or set CPU above 95 and click **Push Custom Metrics**.

**From the terminal:**

“`bash
curl -X POST http://localhost:5020/API/v1/metrics/simulate
curl http://localhost:5020/API/v1/alerts/statistics
curl http://localhost:5020/API/v1/alerts/active
“`

Wait 30–60 seconds for pending timers, then refresh the dashboard. Statistics should show `total` greater than zero.

### Run the automated demo

With the stack already running:

“`bash
./build.sh demo
“`

### Stop and clean up

“`bash
./stop.sh # stop app processes and Docker Compose services
./cleanup.sh # remove node_modules, venv, caches; prune Docker resources
“`

—

## Docker

**Hybrid mode** (what `./build.sh local` uses) — infrastructure only:

“`bash
docker compose up -d postgres redis
“`

The app connects via `localhost:5450` and `localhost:6400`.

**Full stack** — all four services in containers:

“`bash
./build.sh docker
“`

Tear down containers and volumes:

“`bash
docker compose down -v –remove-orphans
“`

—

## Dashboard Walkthrough

The **Live** badge in the header means the WebSocket connection is active. When an alert changes state, the table updates without reloading the page.

—

## API Reference

Base URL: `http://localhost:5020`

### Health check

“`bash
curl -sS http://localhost:5020/API/health | python3 -m json.tool
“`

Expected:

“`json
{“status”: “healthy”, “service”: “infrawatch-alerts”, “rules”: 4}
“`

### Ingest a single metric

“`bash
curl -sS -X POST http://localhost:5020/API/v1/metrics/ingest
-H “Content-Type: application/json”
-d ‘{“metric_name”:”cpu.utilization”,”value”:95.0,”service”:”compute”}’
“`

### Simulate a production burst

“`bash
curl -sS -X POST http://localhost:5020/API/v1/metrics/simulate | python3 -m json.tool
“`

### Query alerts

“`bash
curl -sS http://localhost:5020/API/v1/alerts/statistics | python3 -m json.tool
curl -sS http://localhost:5020/API/v1/alerts/active | python3 -m json.tool
“`

### Acknowledge and test a rule

“`bash
curl -sS -X POST “http://localhost:5020/API/v1/alerts/{alert_id}/acknowledge”

curl -sS -X POST http://localhost:5020/API/v1/test/rule
-H “Content-Type: application/json”
-d ‘{“rule_id”:”“,”test_values”:[95.0, 50.0, 10.0]}’
“`

### Endpoint summary

| Method | Path | Purpose |
|——–|——|———|
| GET | `/API/health` | Health check |
| GET/POST | `/API/v1/rules` | Rule CRUD |
| POST | `/API/v1/rules/bulk` | Bulk enable/disable/delete |
| GET/POST | `/API/v1/templates` | Rule templates |
| POST | `/API/v1/test/rule` | Test rule against sample values |
| POST | `/API/v1/metrics/ingest` | Ingest metric + trigger evaluation |
| POST | `/API/v1/metrics/simulate` | Demo metric burst |
| POST | `/API/v1/alerts/search` | Filtered search with pagination |
| GET | `/API/v1/alerts/statistics` | Dashboard aggregates |
| GET | `/API/v1/alerts/active` | Non-resolved alerts |
| POST | `/API/v1/alerts/{id}/acknowledge` | Acknowledge |
| POST | `/API/v1/alerts/{id}/resolve` | Resolve |
| POST | `/API/v1/alerts/bulk-update` | Bulk status change |
| POST | `/API/v1/alerts/export` | CSV/JSON export |
| WS | `/ws/alerts` | Live alert updates |

—

## Troubleshooting

—

## Assignment

1. Add a composite rule that fires only when **both** `cpu.utilization > 80` AND `memory.utilization > 75`.
2. Create a suppression rule for `payments.*` metrics during a 2-hour maintenance window.
3. Build an escalation policy that notifies Slack at level 1 (5 min) and PagerDuty at level 2 (15 min).
4. Add a dashboard widget showing MTTR trend over the last 24 hours.

### Solution Hints

– Composite rules: extend `RuleType.COMPOSITE` in `evaluator.py` with AND/OR sub-condition evaluation.
– Suppression: POST to `/API/v1/suppression` with regex `metric_pattern` and ISO window timestamps.
– Escalation: attach `EscalationPolicy` rows to rules; background loop checks `delay_minutes` on unacknowledged firing alerts.
– MTTR widget: query `/API/v1/alerts/statistics` hourly and plot `mttr_minutes` in a `LineChart`.

—

*Next in series: Notification channels — email, Slack, webhooks, and preference-based delivery.*

- Hands-On Tutorial

Anjali

Hands-On System Design Tutorial

## Atlas Hyperscale Platform — Engineering Lesson

**Domain:** DevOps · SecOps · SRE · Cloud Engineering · Platform Engineering · Release Engineering · Automation Engineering · Infrastructure Engineering

## What We Build

You will operate **Atlas Hyperscale Platform** — a single control plane that unifies global traffic engineering, predictive scaling, FinOps accountability, and production-readiness gates. When Netflix serves a global catalog or Shopify absorbs a flash-sale spike, no single team owns “the server.” They own **loops**: route traffic cheaply, scale before saturation, spend consciously, and gate launches with evidence. Atlas teaches that integration pattern in one repository.

The platform is built **from scratch as one product**: four domain routers share one FastAPI process, one React console, and one WebSocket broadcast. Traffic, performance, economics, and readiness are first-class modules — not separate repos glued at the UI.

**Agenda**

– **Traffic engineering** — geographic routing, consistent-hash sharding, multi-tier cache, regional failover
– **performance operations** — profiling, predictive auto-scaling, query optimization, capacity runway
– **FinOps governance** — namespace cost attribution, optimization recommendations, anomaly alerting
– **Production readiness** — six-pillar validation, integration tests, runbooks, knowledge base
– **Unified console** — live dashboard with cross-domain **Run Platform Demo**

**Success criteria:** Dashboard shows live metrics; validation returns six pillar scores; FinOps total cost updates after collection; failover reroutes traffic away from a failed region.

## Why a Unified Control Plane Matters

In most organizations, traffic routing lives in CDN configs, scaling in a separate autoscaler service, cost in a spreadsheet, and launch approval in a wiki. Each layer works until something breaks at a boundary — a region fails but FinOps still bills it, autoscaler adds replicas after latency already spiked, or a service ships without a runbook.

Atlas models the **opposite architecture**:

You are not learning four unrelated APIs — you are learning **how platform engineers wire hyperscale loops together**.

## Platform Placement

Atlas sits at the **coordination layer** between product traffic and raw cloud infrastructure — the same role internal platforms play at hyperscale retailers, social feeds, and fintech payment rails.

Atlas does not replace Kubernetes, cloud consoles, or observability backends. It exposes a stable internal API so a product team can request a user profile without knowing which region, shard, or cache tier served it — while SRE and platform teams retain policy control behind that abstraction.

## Core Concepts by Engineering Discipline

### Infrastructure & Cloud Engineering

Hyperscale systems partition work across **regions** (latency), **shards** (data scale), and **cache tiers** (cost). Atlas simulates four regions (`us-east`, `us-west`, `eu-west`, `ap-south`) with independent health scores and capacity. A request carries optional `X-User-Lat` / `X-User-Lon` headers; the engine picks the nearest healthy region by Euclidean distance — the same intuition behind AWS Route 53 latency routing and Cloudflare anycast edge selection.

**Sharding** uses consistent hashing: each physical shard owns 256 virtual nodes on a hash ring. When you add shard five, only ~20% of keys move (idealized) versus naive modulo partitioning that reshuffles half the dataset. Cassandra, DynamoDB, and Redis Cluster all rely on this property.

| Cache tier | TTL | Role |
|————|—–|——|
| L1 | 30 s | Hot user sessions, flash traffic |
| L2 | 5 min | Warm catalog / profile data |
| L3 | 30 min | Stable reference data |
| Miss | — | Origin fetch; counted for hit-rate SLO |

Cloud engineers care because **cache hit-rate is a cost lever**. Every miss is a database round-trip billed in CPU, connection pool slots, and replication lag risk.

### Platform & Release Engineering

Four domain routers share one FastAPI application:

On startup (`lifespan` in `main.py`), Atlas wires router dependencies, collects initial FinOps metrics, and launches background loops: profiler sampling, autoscaler forecasting, query optimizer refresh, capacity runway calculation, and metrics broadcast every second. That bootstrap mirrors real controllers that reconcile desired state after the API process comes up.

Release engineers attach promotion logic to **one surface**: readiness score ≥ threshold, integration pass rate 100%, no active FinOps alerts — before tagging a production release.

### DevOps & Automation Engineering

The **platform demo** (`POST /API/platform/demo`) is an orchestrated simulation: ramp traffic target RPS, bump load metrics, trigger cost collection, and return a unified summary. Automation teams use the same pattern in production — a “game day” script that proves subsystems cooperate before Black Friday.

Background tasks run as asyncio loops, not cron jobs inside the API process. That is intentional: shared in-memory state (replica count, cache stats, cost totals) stays consistent for WebSocket subscribers without a separate message bus in this learning stack.

### SecOps

Readiness validation includes a **security pillar** scoring policy compliance and automation coverage. FinOps anomaly detection uses statistical thresholds (3σ above baseline) — the same class of signal SIEM rules use for billing fraud or crypto-mining hijacks on compromised accounts.

Regional failover is a security-adjacent control: when `us-east` is marked failed, traffic never routes there until recovery — limiting blast radius during regional compromise or BGP incidents.

### SRE

Atlas exposes Prometheus gauges at `/metrics` (`atlas_readiness_score`, `atlas_cluster_cost_usd`, `atlas_validation_runs_total`). The console subscribes to `/ws/live` so operators see golden signals without refreshing.

**Error-budget thinking:** If p99 latency degrades while average CPU looks healthy, suspect cache stampede or shard hotspot — not “add more servers blindly.” The Traffic tab shows per-shard load % exactly for this diagnosis.

### FinOps & Cloud Financial Management

Namespace-level costs (`production`, `staging`, `ml-training`, etc.) enable **chargeback** — each team sees its bill before finance closes the month. Recommendations cover rightsizing, reserved instances, idle volume cleanup, and spot migration. Waste reports aggregate orphaned resources.

Production FinOps teams run this loop weekly: collect → attribute → recommend → alert → remediate. Atlas compresses it into a five-minute demo.

## Architecture, Control Flow & Data Flow

**Control flow:** React console calls REST endpoints and subscribes to `/ws/live`. User actions (fail region, start load, run validation) mutate service state; background loops propagate changes to all connected clients.

**Data flow (user request path):**

“`text
GET /API/traffic/user/{id}
→ parse X-User-Lat, X-User-Lon
→ route_request() → nearest healthy region
→ get_shard(user_id) → consistent-hash ring lookup
→ check_cache() → L1 → L2 → L3 → miss
→ set_cache() on miss
→ return { region, shard, cache_hit, profile }
“`

**Data flow (observability path):**

“`text
background_metrics (1s) ─┐
background_finops (120s) ─┼→ broadcast_ws() → /ws/live → dashboard panels
autoscaler (30s) ─┘
“`

**State transitions:**

## Capability Deep Dive

### Traffic module

The `HyperscaleEngine` maintains region health, shard statistics, and cache counters. Admin endpoints simulate production controls:

“`text
POST /API/traffic/admin/start-load {“target_rps”: 5000}
POST /API/traffic/admin/fail-region/us-east
POST /API/traffic/admin/recover-region/us-east
GET /API/traffic/metrics
“`

**Failover workflow:** fail region → `route_request` skips it → users near NYC route to `us-west` or `eu-west` → shard assignment unchanged (shard is independent of region in this model). Recovery restores optimal latency routing.

**Design note:** Production systems separate **routing** (which edge serves the request) from **data placement** (which shard owns the key). Atlas keeps both visible in one response so you practice reading `region` and `shard` together — a common on-call skill during incidents.

### performance module

The autoscaler uses exponential smoothing (α = 0.2) plus a sinusoidal seasonality factor — a simplified version of what HPA+VPA controllers and KEDA scalers approximate with real metrics pipelines.

“`text
if predicted_utilization > 0.80: scale up (proactive)
if sustained_utilization < 0.40: scale down (conservative) ``` **Insight:** Scale-down waits for sustained low load because premature teardown causes **flapping** — replicas added and removed every few minutes, wasting provisioning API calls and cold-start latency. ### FinOps module ```text POST /API/finops/cost/collect GET /API/finops/cost/total GET /API/finops/cost/breakdown # compute 58% / storage 27% / network 15% GET /API/finops/optimize/recommendations GET /API/finops/optimize/waste GET /API/finops/alerts ``` Cost collection runs on a 120-second background loop and on demand. Anomaly detector may emit alerts when spend spikes above baseline — surfacing in the FinOps tab and WebSocket payload. **Real-world parallel:** Kubernetes cost tools (Kubecost, OpenCost) attribute spend by namespace label; Atlas uses the same mental model with simulated namespace totals. ### Readiness module Six validators run concurrently via `asyncio.gather`: | Pillar | What it measures | |--------|------------------| | Reliability | Failover, redundancy checks | | Security | Policy compliance | | Observability | Metrics and tracing coverage | | Cost | Budget and optimization posture | | scalability | Shard balance, autoscaler health | | Operability | Runbook and knowledge completeness | ```text POST /API/readiness/validate → six scores + overall + recommendations POST /API/readiness/integration-tests → 10 E2E scenarios GET /API/readiness/runbooks GET /API/readiness/knowledge ``` Integration tests cover `auth_flow`, `traffic_routing`, `cache_consistency`, `autoscaler_response`, `cost_collection`, `readiness_gate`, `failover_recovery`, `shard_rebalance`, `metrics_pipeline`, and `runbook_generation` — proving cross-module wiring, not isolated unit behavior. Runbooks include `latency_spike`, `region_failover`, `cost_spike`, `shard_hotspot`, `cache_stampede`, and `scale_event` — each with severity and step count. Knowledge materials support onboarding without hunting tribal wiki pages. ## Console Behavior The React dashboard uses five tabs: | Tab | Primary data source | Key actions | |-----|---------------------|-------------| | Overview | `/API/platform/summary` + WebSocket | Run Platform Demo | | Traffic | `/API/traffic/metrics` | Start load, fail/recover region | | performance | autoscaler, runway, optimizer | View forecast confidence | | FinOps | cost total, breakdown, recommendations | View active alerts | | Readiness | validation, runbooks, knowledge | Run validation, integration tests | WebSocket payload bundles traffic, performance, finops, and readiness in one JSON frame — the pattern production consoles use to avoid four polling intervals fighting each other. ## Context in Distributed Systems Atlas is a **teaching compression** of patterns seen at: - **Global edge networks** (Cloudflare, Fastly) — geographic routing and cache tiers - **Hyperscale data planes** (DynamoDB, Cassandra) — consistent hashing and shard rebalancing - **E-commerce flash events** (Shopify, Amazon Prime Day) — predictive scaling ahead of demand - **Cloud FinOps maturity** (AWS Cost Explorer, Finout) — namespace attribution and anomaly detection - **Production readiness reviews** (Google PRR, Meta readiness templates) — multi-pillar scorecards before launch None of these companies run exactly this stack. They all run **the same loops** with different implementations. Atlas makes the loops visible in one afternoon. ## Assignment 1. Add a fifth region (`ap-northeast`) with Tokyo coordinates; verify routing from headers `X-User-Lat: 35.6`, `X-User-Lon: 139.7`. 2. Implement a cache-stampede runbook scenario and surface it in the Readiness tab. 3. Tune autoscaler `scale_up_threshold` to 0.75; document replica changes under a 2× load spike via `/API/performance/load/generate`. 4. Add a FinOps chargeback tag `team=checkout` on the `production` namespace cost entry. 5. Achieve overall readiness score ≥ 85% and integration test pass rate 100% in one session. ## Solution Hints 1. Extend `HyperscaleEngine.regions` dict; `route_request` picks minimum distance automatically — no router change needed. 2. Add `"cache_stampede"` to `RunbookGenerator.SCENARIOS`; call `POST /API/readiness/runbooks/generate?scenario=cache_stampede`. 3. Edit `PredictiveAutoscaler.scale_up_threshold` in `autoscaler.py`; POST `{"pattern":"spike"}` to load endpoint; watch `/API/performance/autoscaler/status`. 4. Enrich `CostCollector.namespace_costs` values with a `tags` dict in the API response layer of `finops.py`. 5. Run validation twice (scores are stochastic 78–96); integration tests retry until pass or seed `random` in tests for determinism during development. ## Takeaway Production hyperscale is not one big server — it is **four cooperating loops**: route traffic cheaply, scale before saturation, spend consciously, and gate launches with evidence. Atlas gives you a working miniature of that loop. The skill you carry into your first platform team is not memorizing endpoints — it is recognizing which loop is broken when the dashboard turns red, and knowing which module owns the fix.

- Hands-On Tutorial

Anjali

Hands-On System Design Tutorial

## Nexus AI Operations Platform — Engineering Lesson

**Domain:** DevOps · SecOps · SRE · Cloud Engineering · Platform Engineering · Release Engineering · Automation Engineering · Infrastructure Engineering

## What We Build

You will design and operate **Nexus AI Operations Platform** — a single control plane that treats specialized accelerators, edge inference nodes, ML-driven infrastructure decisions, secure delivery tooling, responsible-AI governance, and autonomous remediation as one coherent system rather than a collection of disconnected tools.

The platform is built **from scratch as one product**: every module shares the same API process, the same console, and the same operational vocabulary. There are no separate mini-projects stitched together at the UI — compute, edge, intelligence, governance, and automation are first-class domains inside one repository.

**Agenda**

– **Compute orchestration** — GPU multi-instance sharing, placement policies, and spend visibility
– **Accelerator scheduling** — TPU pod queues with preemptible cost optimization
– **Edge fleet management** — device registration, model rollout, heartbeat health, cloud sync
– **Infrastructure intelligence** — forecasting, anomaly detection, predictive scaling, incident response
– **Delivery intelligence** — code security analysis, log anomaly detection, alert correlation, documentation generation
– **Model governance** — bias analysis, fairness monitoring, explainability, approval workflows
– **Autonomous operations** — DAG workflows, self-healing, controlled chaos, AI-assisted scheduling
– **Unified console** — real-time dashboard with cross-domain **Run Demo** simulation

**Platform objective:** Deliver the integration pattern production ML platform teams use before opening a shared GPU pool to multiple product lines — one API boundary, explicit promotion gates, observable state, and automation that closes the loop.

## Why a Unified Control Plane Matters

In most organizations, GPU scheduling lives in one team’s scripts, edge rollout in another’s Ansible playbooks, bias review in a spreadsheet, and incident correlation in a third-party SaaS. Each layer works in isolation until something breaks at a boundary — a model promoted without review, inference scaled without reading logs, or a TPU job submitted onto full-GPU capacity because nobody exposed MIG slices.

Nexus models the **opposite architecture**: one FastAPI application registers seven domain routers; one React console polls them; one demo orchestrator proves they cooperate. That shape mirrors how mature internal platforms (not public cloud consoles) are actually operated:

You are not learning seven unrelated APIs — you are learning **how platform engineers wire them together**.

## Platform Placement in the Overall System

Modern AI infrastructure at scale (recommendation training at Meta, TPU fleets at Google, factory-floor inference at industrial operators) converges on three planes that must cooperate:

Nexus is the **coordination layer** between product engineers and raw infrastructure. It does not replace Kubernetes, cloud consoles, or observability backends. It exposes a stable internal API so an inference team can submit a job without knowing whether it lands on a MIG slice, a preemptible TPU pod, or an edge CPU node — while SRE and platform teams retain policy control behind that abstraction.

## Core Concepts by Engineering Discipline

### Infrastructure & Cloud Engineering

Accelerator capacity is a **scarce, billed state machine**. NVIDIA MIG (Multi-Instance GPU) partitions one physical GPU into isolated instances with fixed memory profiles. Common profiles on A100-class hardware include:

| Profile | Approx. memory | Typical use |
|———|—————-|————-|
| `1g.5gb` | 5 GB | Small inference, embedding models |
| `2g.10gb` | 10 GB | Medium inference batches |
| `3g.20gb` | 20 GB | Larger single-model serving |
| `7g.40gb` | 40 GB | Near full-GPU workloads |

MIG is not “fractional sharing” in the sense of best-effort CPU quotas — each instance has **hardware-enforced isolation**. That is why platform teams prefer MIG for multi-tenant inference: one bad neighbor cannot exhaust another tenant’s memory budget.

TPU pods are similarly finite; preemptible instances trade restart risk for sharply lower hourly rates. A training job that checkpoints every N steps is a natural preemptible candidate; a latency-sensitive inference path is not.

The Nexus scheduler encodes placement economics:

“`text
if memory_gb < 10: strategy = mig elif checkpoint_enabled: strategy = spot else: strategy = full_gpu ``` Cloud engineers care because misplacement appears on the invoice weeks before it appears in latency dashboards. Cost tracking per resource type (`on_demand`, `spot`, `mig_slice`) makes FinOps visible at scheduling time, not only at month-end reconciliation. **Production parallel:** Cloud providers expose MIG via drivers and device plugins; Nexus simulates inventory and allocation so you practice the **policy layer** without bare-metal GPUs. ### Platform & Release Engineering Seven domain routers share one FastAPI application: | Router prefix | Domain | |---------------|--------| | `/API/compute/gpu` | MIG, scheduling, cost | | `/API/compute/tpu` | Job queue, pod matching | | `/API/edge` | Fleet, devices, deploy, sync | | `/API/infra` | Forecast, anomaly, scale | | `/API/devops` | Code, logs, incidents, docs | | `/API/governance` | Bias, fairness, explain, reviews | | `/API/automation` | Workflows, heal, chaos, schedule | | `/API/platform` | Summary + cross-domain demo | Each exposes a versioned REST contract. Release engineers benefit because promotion logic — bias check passed, governance approved, chaos recovery verified — can attach to **one platform surface** regardless of deployment target. On application startup (`lifespan` in `main.py`), Nexus: 1. Initializes SQLite tables for governance and bias records 2. Starts background subsystems (self-healing, chaos framework, AI scheduler) 3. Seeds a demo edge fleet 4. Submits a bootstrap workflow DAG (“Validate GPU Inventory” → “Sync Edge Fleet”) That bootstrap sequence mirrors real controllers that reconcile desired state after the API process comes up. ### DevOps & Automation Engineering Security analysis runs against abstract syntax trees before merge, catching patterns like `eval()`, hardcoded credentials, and unsafe deserialization. Log streams feed an isolation-forest detector; correlated alerts collapse into a single incident via density clustering (DBSCAN) rather than five separate pages. Workflow engines execute DAGs asynchronously; chaos experiments validate that recovery paths actually work. Automation design principle: **idempotent remediation** — restart, scale, reschedule — not runbooks that assume a human is awake. The automation router also exposes a WebSocket for live workflow status — useful when operators want push updates instead of polling. ### SecOps Correlation before paging is the operational difference between a healthy on-call rotation and burnout. Nexus groups alerts that share temporal and source proximity into one incident object with an inferred root cause string. Secret-pattern scanning in the delivery path is cheaper than credential rotation after exfiltration. When you `POST` code containing `password="..."` to the analyzer, the score drops and severity-tagged issues return — that is the same class of signal CI secret scanners emit, implemented locally for learning. ### SRE Time-series forecasting on CPU, memory, and request-rate history enables scale-up **before** saturation. Anomaly detection flags metric spikes that precede SLO breach. Prometheus-compatible GPU metrics at `/metrics` allow the same observability toolchain to scrape accelerator posture alongside application SLIs. Error-budget thinking applies to GPU pools: if inference latency degrades while average utilization looks healthy, the scheduler policy — not traffic — is the likely culprit. **Golden signals in Nexus context:** | Signal | Where observed | Action triggered | |--------|----------------|------------------| | Latency | Log ingest + incident | Correlated alert | | Traffic | Metric history | Forecaster train/predict | | Errors | Log level + anomaly score | Incident create | | Saturation | CPU/memory % | Scaling agent evaluate | ### Release Engineering Governance is a **deployment gate**, not a spreadsheet. Bias analysis computes demographic parity and equalized-odds spreads; failed models do not receive approval records. Fairness monitoring logs per-group outcome rates over time. Explainability returns feature-attribution weights for audit trails. The fair demo model `loan-model-v1` is designed to pass parity checks; other model IDs in the simulator may fail — practice reading metric spreads, not just boolean pass/fail. ## Capability Deep Dive ### GPU compute module The MIG controller maintains simulated A100 inventory (four GPUs in demo), enables partitioning per GPU, configures profile/instance counts, and allocates slices to workloads. **Typical API sequence:** ```text GET /API/compute/gpu/inventory POST /API/compute/gpu/{id}/mig/enable POST /API/compute/gpu/{id}/mig/configure # body: profiles + instance counts POST /API/compute/gpu/schedule # body: name, memory_gb, checkpoint_enabled GET /API/compute/gpu/cost/analytics GET /metrics # Prometheus text with gpu_utilization ``` The scheduler creates jobs with strategy metadata; the cost module tracks hourly spend and emits migration recommendations when on-demand hours accumulate (e.g., suggest spot or MIG partitioning). **Workflow:** enable MIG → configure profiles → schedule job → allocate slice → record cost → expose Prometheus gauges. **Design note:** Scheduling is synchronous in the demo — production systems enqueue to a controller that reconciles against real node capacity. The **policy function** (MIG vs spot vs full) is what you carry forward. ### TPU compute module A resource manager holds `v4-8` and `v4-32` pods with preemptible flags. Jobs enter a queue; the orchestrator matches type and preemptible preference to available pods. Cost estimation aggregates running jobs into hourly and daily projections with preemptible savings. **Submit payload concepts:** | Field | Role | |-------|------| | `name` | Human-readable job identifier | | `tpu_type` | Pod shape (`v4-8`, `v4-32`) | | `prefer_preemptible` | Cost policy — prefer cheaper interruptible capacity | **Insight:** Preemptible preference is a **cost policy knob**, not an afterthought — it belongs in the submit payload, not in a post-hoc FinOps report. Cluster metrics (`/API/compute/tpu/metrics/cluster`) expose utilization % and pod counts — the same numbers the overview card displays after **Run Demo**. ### Edge fleet module Devices register with name, location, and capability tags (`gpu`, `inference`, `cpu`). Heartbeats refresh online state and CPU/memory telemetry. Model deployment uses capability selectors so inference models reach appropriate nodes. Sync metrics report bytes transferred during fleet reconciliation. **Device lifecycle:** ```text register → online ←── periodic heartbeat ──→ offline (TTL expired) └── deploy model (selector match) └── sync_metrics (bytes reconciled) ``` **Distributed systems note:** Heartbeat TTL marks devices offline without blocking the control plane — partition tolerance by design. The fleet summary (`online` / `total`) appears in `/API/platform/summary` for the overview panel. ### Infrastructure intelligence module Metrics history feeds a RandomForest forecaster for short-horizon prediction. IsolationForest marks statistical outliers. A scaling agent compares predictions to thresholds and emits scale-up/down decisions per logical deployment (`API-service`, `worker-pool`, `inference-gateway`). An incident response agent maps recent anomalies to remediation action lists. **Data flow:** ```text record(metric, value) → train(forecaster) → predict(next N steps) → detect(anomaly) → create incident → suggest remediation ``` After **Run Demo**, random metric spikes populate history, trigger training, and feed `scaling_agent.evaluate()` — you should see CPU utilization move on the overview card. ### DevOps intelligence module | Capability | Endpoint pattern | Output | |------------|------------------|--------| | Code security | `POST /API/devops/code/analyze` | Issues + score | | Log anomaly | `POST /API/devops/logs/ingest` | Ingested + anomaly flag | | Alert correlation | `POST /API/devops/incidents/alerts` | Clustered incident | | Doc generation | `POST /API/devops/docs/generate` | Markdown API summary | The demo runner ingests synthetic error logs and creates alerts so the DevOps panel shows non-empty state after one click. ### Governance module Bias detector computes parity and odds metrics against synthetic cohorts; results persist to SQLite (`data/nexus.db`). Fairness monitor aggregates per-group outcome rates. Explainability service returns normalized feature-importance maps. Governance service manages submit → pending → approved/rejected lifecycle. **Review workflow:** ```text POST /API/governance/reviews/submit → status: pending_review POST /API/governance/reviews/{id}/approve POST /API/governance/reviews/{id}/reject GET /API/governance/reviews/stats ``` Persistence here is intentional — unlike ephemeral GPU job state, audit records must survive process restarts. ### Automation module | Subsystem | Responsibility | |-----------|----------------| | Orchestration engine | Async DAG task execution | | Self-healing controller | Resource registration + remediation log | | Chaos framework | Typed failure injection + recovery flag | | AI scheduler | Predicted optimal workflow windows | Chaos types include `pod_failure`, `network_latency`, and `cpu_stress`. Injecting `pod_failure` returns `"recovered": true` when the healing path succeeds — that is your proof that automation is wired, not just listed in a diagram. ### Platform demo orchestrator `POST /API/platform/demo` runs a coordinated simulation across all modules. Internally, `demo_runner.py` sequences roughly: 1. **GPU** — enable/configure MIG on a random GPU; schedule 1–3 jobs with random memory and checkpoint flags; record cost usage 2. **TPU** — submit 1–2 training jobs with preemptible preference 3. **Edge** — heartbeat every registered device; run fleet sync 4. **Infra** — spike all metric series; train forecaster; evaluate scaling; analyze anomalies 5. **DevOps** — ingest error log; create correlated alert 6. **Governance** — run bias analysis; submit governance review 7. **Automation** — inject chaos; submit workflow with dependent tasks The response is a **unified snapshot** (GPU active jobs, TPU utilization %, edge online count, infra CPU %) — the same shape as `GET /API/platform/summary`, which the console polls every few seconds. **Why one button:** Production validation often requires a “synthetic canary” that touches every dependency. The demo endpoint is that canary for the integrated platform. --- ## Operations Console The React dashboard (`frontend/src/App.jsx`) uses a light theme with color-coded stat cards — indigo for GPU, violet for TPU, cyan for edge, emerald for CPU/health. **Navigation panels:** | Panel | Primary data sources | |-------|---------------------| | Overview | `/API/platform/summary` | | Compute | GPU inventory, TPU jobs, cost endpoints | | Edge Fleet | devices, fleet status, sync | | Infra AI | dashboard stats, model train, scaling | | DevOps AI | code analyze, logs, incidents | | Governance | bias, fairness, reviews | | Automation | workflows, healing log, chaos | **Run Demo** calls `POST /API/platform/demo`, shows a toast on success, and briefly flashes updated card values. Without running demo first, overview text reads *“Click Run Demo to simulate live workloads”* — the platform is healthy but idle. Local development runs Vite on port 3000 with API proxy to 8000. Docker serves the built `dist/` via nginx on the same port. --- ## Context in Distributed Systems ### Control plane vs data plane | Layer | Nexus implementation | Production analogue | |-------|---------------------|---------------------| | Control | Routers, schedulers, governance gates | K8s controllers, internal platform APIs | | Data | Simulated jobs, heartbeats, metric series | Real training jobs, edge agents, Prometheus | | Observation | Dashboard, `/metrics`, demo runner | Grafana, PagerDuty, FinOps tooling | ### Why integration matters ML platforms fail at **handoffs**: training finishes without bias review; inference autoscales without correlating log anomalies; edge models deploy without governance records. Nexus wires these handoffs into one codebase so you practice production integration, not isolated tutorials. ### Consistency and availability trade-offs | State type | Storage | Rationale | |------------|---------|-----------| | GPU/TPU jobs, metrics, workflows | In-memory | Fast demo + test isolation | | Bias results, governance reviews | SQLite | Audit persistence | | Edge fleet | In-memory + seed on startup | Partition-tolerant heartbeats | Edge fleet state tolerates stale heartbeats — the scheduler remains available even when edge links flap. This is **AP** leaning: availability over strict fleet consistency. --- ## Architecture ### Request path (example: schedule GPU job) ```text Browser or curl → POST /API/compute/gpu/schedule → gpu router validates body → gpu_scheduler_service.schedule(...) → returns { id, strategy, status } → console polls inventory / summary ``` ### Control flow 1. Operator interacts via console or API client. 2. Router validates payload and delegates to a domain service. 3. Service mutates state and returns JSON. 4. Console polls summary and domain endpoints every 3–5 seconds. 5. **Run Demo** triggers `demo_runner`, which sequences cross-domain side effects in one request. ### State machines **Compute job:** ```text submitted → scheduled → running → completed └──────→ failed → healing → rescheduled ``` **Governance:** ```text not_submitted → pending_review → approved | rejected ``` **Edge device:** ```text online ←── heartbeat ──→ offline (TTL expired) ``` --- ## Technology Choices | Layer | Choice | Why | |-------|--------|-----| | API | FastAPI 3.11 | Async lifespan, OpenAPI docs at `/docs`, Pydantic validation | | ML utilities | scikit-learn | Forecaster + isolation forest without heavy DL deps | | Persistence | SQLite | Zero-config audit store for governance | | Metrics | prometheus_client | Industry-standard scrape format | | UI | React + Vite | Fast dev loop, static production build | | Containers | Compose | One command for full stack | ## Success Criteria - Integration test suite passes from project root - Console renders white theme with color-coded metric cards - **Run Demo** updates live values and shows confirmation toast - Fair model bias check returns `passed: true` - Chaos experiment reports `recovered: true` - `/metrics` exposes GPU Prometheus text ## Assignment Extend the integrated platform (not separate repos): 1. Add a **unified cost panel** combining GPU spend and TPU hourly estimates with a 7-day projection. 2. Auto-trigger bias analysis when a TPU job payload includes `model_id`; block queue if `passed` is false. 3. Fire a DevOps incident when two or more edge devices go offline within the heartbeat window. 4. Chaos-fail a workflow task and verify self-healing restarts the dependent resource. ## Solution Hints 1. New route `/API/platform/cost-summary` aggregating GPU analytics and TPU cost estimate; poll every 30s in React. 2. In TPU submit handler, call bias analyzer before allocation; return 403 if not passed. 3. In edge offline marking logic, count stale devices; if ≥ 2, create correlated DevOps incident. 4. Submit workflow with failing task; invoke healing controller on failure event from incident detector. ## Practical Takeaway Specialized hardware without intelligent operations burns budget silently. Nexus demonstrates how platform teams unify accelerators, edge nodes, ML-driven ops, security signals, ethics gates, and autonomous remediation behind **one control plane** — the shape of system you ship before ML workloads share expensive silicon. The skills transfer directly: - **Scheduler policy** → real cluster autoscalers and quota systems - **Governance gates** → model registry promotion workflows - **Demo orchestrator** → synthetic monitoring and chaos game days - **Unified summary API** → single pane of glass for leadership and on-call Build the integration muscle here; swap simulated backends for real cloud APIs when you deploy to production.

- Hands-On Tutorial

Anjali

Hands-On System Design Tutorial

## Lesson 7 — Unified MLOps Control Plane

## What We’re Building Today

Today you build a **unified MLOps control plane** — a single platform that orchestrates the entire machine learning lifecycle: experiment tracking, data pipelines, distributed training, model serving, production monitoring, security governance, and cost optimization.

This is not a collection of disconnected notebooks. It is an **integrated system** with a shared API, centralized state, real-time observability, and a command-center dashboard — the kind of architecture platform teams deploy when ML stops being a research exercise and becomes a revenue-critical production capability.

### Why It Matters

Most organizations fail at MLOps not because they lack tools, but because their tools are **siloed**:

– Data engineers own pipelines in Airflow.
– ML scientists own experiments in scattered notebooks.
– Platform engineers own Kubernetes clusters nobody else understands.
– FinOps owns a spreadsheet disconnected from GPU utilization.
– Security owns a governance process that blocks releases at 4 PM on Fridays.

A unified control plane collapses these silos behind **one API surface**, **one state model**, and **one operational vocabulary**. Netflix’s Metaflow, Uber’s Michelangelo, and Spotify’s ML infrastructure all converge on this pattern: abstract complexity, expose capabilities, reconcile state continuously.

### Practical Outcome

By the end of this lesson you will have:

– A **FastAPI control plane** exposing seven domain modules through REST and WebSocket APIs.
– **Real ML training** with MLflow experiment tracking and scikit-learn model registration.
– A **live dashboard** with module navigation, metrics, and activity streaming.
– **Docker-packaged deployment** with health checks, Prometheus metrics, and persistent runtime volumes.
– **Integration tests** validating the full pipeline from ingest to inference to drift detection.

### High-Level Agenda

– Understand the MLOps lifecycle as a distributed control plane problem
– Design module boundaries: platform, pipeline, training, serving, monitoring, security, cost
– Implement shared state reconciliation and event broadcasting
– Wire real services: MLflow, sklearn, Prometheus, WebSocket feeds
– Containerize API and UI with Docker Compose
– Validate end-to-end with integration tests and production checklists
– Extend the platform with advanced patterns: operators, GitOps, multi-cluster serving

## Core Concepts: The MLOps Lifecycle

### What is MLOps?

MLOps is the **operational discipline** of taking machine learning from experiment to production and keeping it there reliably. It applies the same engineering rigor that transformed software deployment — version control, CI/CD, observability, incident response — to the unique challenges of ML systems.

ML systems are not ordinary applications. They have **three axes of change** that traditional DevOps was never designed for:

1. **Code** — training scripts, feature logic, serving handlers
2. **Data** — distributions shift silently; yesterday’s training set is today’s liability
3. **Models** — artifacts with versioned weights, hyperparameters, and performance envelopes

When any axis changes independently, production behavior drifts. MLOps exists to make that drift **visible, measurable, and recoverable**.

### Why This Matters

A model that scores 94% accuracy in a notebook is worthless in production if:

– Nobody can reproduce the training run
– The serving endpoint has no traffic splitting for safe rollout
– Drift goes undetected for three weeks
– GPU costs triple because nobody tracks spot utilization
– A governance workflow was never approved before deployment

Production MLOps is not about tools. It is about **contracts**: between teams, between systems, and between desired state and actual state.

### Real Production Context

At Uber, Michelangelo handles feature stores, training orchestration, and batch/online serving behind unified APIs. At Airbnb, ML platforms enforce that every model has lineage, monitoring, and rollback paths before receiving production traffic. At Spotify, squad-level autonomy is balanced against platform-wide standards for experiment tracking and model registry access.

The pattern is consistent: **platform team builds the control plane; product teams consume capabilities through APIs and dashboards.**

—

## Core Concepts: Control Plane vs Data Plane

### What is the Control Plane?

In distributed systems terminology, the **control plane** makes decisions: what should run, where, and under what policy. The **data plane** executes work: moving bits, running inference, processing batches.

In Kubernetes, the API server and controllers are the control plane; kubelet and container runtime are the data plane. In our MLOps platform:

### Why This Matters

Mixing control and data plane concerns creates systems that are impossible to scale or reason about. If your API server also runs GPU training synchronously, a single long training job blocks health checks, metrics scraping, and governance approvals.

Separating them allows:

– **Independent scaling** — scale inference replicas without scaling the control API
– **Failure isolation** — a crashed training worker does not take down the registry
– **Policy enforcement** — governance checks happen in the control plane before data plane work is scheduled

### Real Production Context

Netflix’s ML platform separates experiment orchestration (control) from actual Spark/Metaflow job execution (data). The control plane tracks desired state; workers report actual state. This is the same reconciliation pattern Kubernetes uses — and we implement a simplified version in `mlops_state.py`.

—

## Core Concepts: Experiment Tracking and Model Registry

### What is Experiment Tracking?

Experiment tracking records **every training attempt** as a structured run: hyperparameters, metrics, artifacts, lineage, and environment. MLflow is the de facto open-source standard, used by thousands of teams and integrated into Databricks, AWS SageMaker, and Azure ML.

A training run without tracking is technical debt. Six months later, nobody knows which data version, which hyperparameters, or which random seed produced the model currently serving 40% of production traffic.

### Why This Matters

The model registry closes the loop between experimentation and deployment. A registered model has:

– Version history
– Stage transitions (Staging → Production → Archived)
– Lineage to the exact MLflow run that produced it

This is the ML equivalent of **immutable container image tags** in CI/CD — you never deploy “latest”; you deploy a specific, auditable version.

### Real Production Context

Spotify teams use experiment tracking to enforce that production models must originate from approved runs with minimum metric thresholds. The registry becomes the **source of truth** for what is allowed to receive traffic — mirroring how container registries gate what images can deploy to production clusters.

—

## Core Concepts: Data Pipeline Integrity

### What is Pipeline Integrity?

ML data pipelines must guarantee **reproducibility** and **quality** before data touches a training job. This means:

– **Ingestion** with schema validation and record counts
– **Validation** with statistical quality scores
– **Versioning** so every training run references an immutable dataset snapshot

Tools like DVC, Delta Lake, and feature stores exist because “we used the CSV from the shared drive” is not a lineage story that passes an audit.

### Why This Matters

Garbage in, garbage out scales linearly with cluster size. A distributed training job on poisoned data wastes GPU hours, produces a confidently wrong model, and creates a monitoring alert three weeks later when business metrics collapse.

Pipeline health metrics — records ingested, validation score, version count — are **leading indicators**. Model accuracy is a **lagging indicator**. Production teams monitor both.

### Real Production Context

Airbnb’s data quality frameworks block downstream training when validation scores drop below thresholds — the same “fail closed” philosophy used in security gateways. Pipeline health status (`healthy`, `degraded`, `failed`) drives automated decisions about whether training jobs may proceed.

—

## Core Concepts: Model Serving and Traffic Management

### What is Canary Deployment for Models?

Deploying a new model version to 100% of traffic on day one is reckless. **Canary releases** route a small percentage of traffic to the new version while monitoring latency, error rate, and business metrics on the incumbent.

Our serving module implements **A/B traffic splitting** between model versions (v1/v2), tracking P99 latency and prediction counts per version.

### Why This Matters

A model can have excellent offline metrics and terrible online performance due to training-serving skew, feature pipeline differences, or adversarial input patterns not present in training data. Gradual rollout limits blast radius.

### Real Production Context

Uber’s serving infrastructure supports shadow mode (new model receives traffic but responses are discarded), canary mode (partial traffic), and full promotion — the same progression Kubernetes Ingress controllers use for application rollouts. The control plane records traffic percentages as **desired state**; the inference router enforces **actual routing**.

—

## Core Concepts: Drift, Fairness, and Observability

### What is Model Drift?

**Data drift** occurs when production input distributions diverge from training data. **Concept drift** occurs when the relationship between features and labels changes. Both degrade model performance silently — accuracy dashboards look fine until they don’t.

Drift detection uses statistical tests (KS test, PSI, population stability) on logged predictions and features. Our monitoring module tracks drift events, 24-hour accuracy, fairness disparity, and explainability request volume.

### Why This Matters

A fraud model trained on pre-pandemic transaction patterns will fail when consumer behavior shifts. Without drift monitoring, the model keeps serving predictions with confidence scores that no longer mean anything.

Fairness monitoring ensures model behavior does not systematically disadvantage protected groups — increasingly a **regulatory requirement**, not a nice-to-have.

### Real Production Context

Netflix monitors model performance across content categories and regions. Spotify tracks recommendation fairness across listener demographics. These are not academic exercises — they are production SLOs with paging policies attached.

—

## Core Concepts: ML Security and Governance

### What is ML Governance?

ML governance is the **approval workflow** that must complete before a model reaches production: security validation, bias analysis, compliance scoring, and immutable audit trails.

Our security module implements:

– **Input validation** — adversarial pattern detection on feature vectors
– **Governance workflows** — submit → review → approve state machine
– **Audit chain** — append-only event log with hash-linked integrity

### Why This Matters

A compromised model endpoint is an attack surface. Adversarial inputs can extract training data, cause misclassification, or trigger resource exhaustion. Governance ensures human accountability — someone approved this model, and we can prove who and when.

### Real Production Context

Financial services and healthcare organizations require audit trails for every model deployment. The EU AI Act and similar regulations are making governance workflows mandatory infrastructure, not optional process.

—

## Core Concepts: FinOps for ML

### What is ML Cost Optimization?

GPU instances are expensive. Spot/preemptible instances are cheaper but interruptible. ML FinOps tracks:

– Daily spend by team and instance type
– Spot utilization percentage
– Savings from optimization recommendations
– ROI on platform investments

Training a model that costs $50,000 in GPU time to improve accuracy by 0.1% when the business impact is $500/month is a failure of **cost-aware engineering**.

### Why This Matters

Cloud bills for ML workloads routinely surprise organizations. Without cost visibility at the control plane level, teams optimize accuracy indefinitely while FinOps discovers the damage in quarterly reviews.

### Real Production Context

Companies running large-scale training on AWS, GCP, or Azure use spot instances, autoscaling, and right-sizing recommendations — the same patterns our cost module simulates with spot utilization tracking and ROI analytics.

—

## Architecture Deep Dive

The platform follows a **modular monolith** architecture: one deployable unit with clear internal boundaries. This is the right starting point for platform teams — simpler than microservices, more structured than a script folder.

### Control Plane Layer

The FastAPI application (`backend/app/main.py`) is the single entry point for all operations:

– **Route registration** — domain-based paths (`/API/platform/*`, `/API/pipeline/*`, etc.)
– **Request validation** — Pydantic models enforce schema contracts
– **Middleware** — Prometheus counters and histograms on every HTTP request
– **WebSocket hub** — broadcasts training completions, drift alerts, cost ticks to connected dashboards
– **Lifespan management** — seeds initial state and starts background tick loop on startup

Module builders (`backend/app/modules/*.py`) translate domain operations into API responses with consistent `{ title, metrics, events }` shapes for the dashboard.

### Data Plane Layer

Services in `backend/app/services/` execute actual work:

### Orchestration Layer

`mlops_state.py` is the platform’s **etcd-equivalent** — centralized state with per-module metric syncers:

– `MLOpsState` dataclass holds all cross-module counters and event lists
– `SYNCERS` map module IDs to metric projection functions
– `record_event()` maintains bounded event buffers for dashboard feeds
– `apply_demo_module()` / `apply_full_demo()` simulate lifecycle progression for demonstrations

The background `_mlops_tick()` coroutine broadcasts state snapshots every 8 seconds — analogous to a Kubernetes controller resync interval.

### Interaction Flow

1. Dashboard or CLI sends `POST /API/platform/train`
2. Control plane validates `TrainRequest` via Pydantic
3. `platform.submit_training()` delegates to `ml_trainer.train_model()`
4. MLflow creates experiment, logs params/metrics, registers artifact
5. `mlops_state` counters increment; event recorded in `training_jobs`
6. WebSocket broadcasts `training_complete` to all connected clients
7. Dashboard metrics grid re-renders with updated counts

### Failure Handling

– Invalid module IDs return `{“error”: “invalid module”, “valid”: […]}` — fail with guidance, not silent 404
– WebSocket clients that disconnect are pruned from `_ws_clients` on send failure
– Docker health checks gate UI container startup until API reports healthy
– `stop.sh` frees ports 8080/3000 before mode switches prevent address-in-use failures

## Control Flow

The end-to-end ML lifecycle executes in this order:

1. **Bootstrap** — `bootstrap.sh` creates venv, installs dependencies, runs integration tests, optionally builds Docker images
2. **Start** — `start.sh` (local) or `start.sh docker` brings up API + dashboard
3. **Data ingest** — `POST /API/pipeline/ingest` generates validated records, updates pipeline health
4. **Data validation** — `POST /API/pipeline/validate` computes quality score; blocks downstream if degraded
5. **Dataset versioning** — `POST /API/pipeline/version` creates immutable snapshot reference
6. **Model training** — `POST /API/platform/train` runs real sklearn training with MLflow tracking
7. **GPU job submission** — `POST /API/training/jobs` queues distributed training with priority and GPU allocation
8. **Governance approval** — `POST /API/security/governance/submit` → `approve` before production promotion
9. **Traffic split** — `POST /API/serving/traffic-split` sets canary percentage
10. **Inference** — `POST /API/serving/predict` routes through A/B split, records latency
11. **Monitoring** — `POST /API/monitoring/log` + `POST /API/monitoring/drift` detect distribution shift
12. **Cost collection** — `POST /API/cost/collect` aggregates spend; optimization endpoints recommend savings
13. **Full demo** — `POST /API/demo/run` exercises all modules in one coordinated pass

## Data Flow

### Input

– **HTTP JSON** — structured requests via REST (`TrainRequest`, `PredictRequest`, etc.)
– **WebSocket** — persistent connection for real-time feed (`/ws`)
– **Synthetic data** — `ml_trainer` generates reproducible datasets for training demonstrations
– **Pipeline data** — ingested records stored in `runtime/pipeline_data`

### Processing

– **Validation layer** — Pydantic models reject malformed requests before service invocation
– **Training pipeline** — features → train/test split → model fit → metric computation → MLflow log
– **Inference pipeline** — instances → version router → model selection by traffic % → predictions + latency
– **Drift pipeline** — logged features → statistical test → severity classification → alert counter
– **Governance pipeline** — workflow submission → state transitions → audit chain append

### Transformations

– Raw feature vectors → sklearn predictions
– Training metrics → MLflow run parameters and artifacts
– Module operations → normalized metric dictionaries via `SYNCERS`
– Events → bounded lists with timestamps and UUIDs for dashboard tables

### Outputs

– **REST responses** — JSON with status, metrics, and domain-specific payloads
– **WebSocket events** — `training_complete`, `drift_check`, `demo_complete`, `mlops_tick`
– **Prometheus metrics** — `mlops_http_requests_total`, `mlops_http_request_duration_seconds`
– **MLflow artifacts** — model binaries and run metadata in `runtime/mlruns`

### State Storage

—

## State Changes & Reconciliation

### Resource Lifecycle

Every platform resource follows a defined lifecycle:

### Desired vs Actual State

– **Desired**: traffic split set to 80/20 via `POST /API/serving/traffic-split`
– **Actual**: `inference_router` routes predictions according to stored percentages
– **Desired**: GPU job with priority 9 submitted
– **Actual**: orchestrator queue position and allocation reflected in `training_queue`

In a full Kubernetes deployment, a ModelServe Custom Resource would declare desired replicas and traffic weights; a controller would reconcile actual pod state. Our in-process state model demonstrates the same **declarative intent → observed state** pattern at application scale.

### Reconciliation Loops

The `_mlops_tick()` background task runs every 8 seconds:

1. Read current cost and drift state from `MLOpsState`
2. Broadcast snapshot to all WebSocket clients
3. Dashboard updates activity stream without polling

This is a simplified **watch/resync** loop — the same primitive that powers Kubernetes informers and Prometheus scrape intervals.

### Drift Detection

When `check_drift()` computes a p-value below threshold:

1. `drift_alerts` counter increments
2. Event appended to `drift_events` list
3. WebSocket broadcasts `drift_check` event
4. Dashboard metric card highlights changed value

Production systems attach **automated responses**: retrain triggers, traffic rollback, or paging on-call ML engineers.

### Self-Healing

– Docker Compose `depends_on: condition: service_healthy` ensures UI starts only after API health check passes
– `stop.sh` kills orphaned processes on ports 8080/3000 before restart
– WebSocket dead client pruning prevents memory leaks from stale connections

### Failure Recovery

– `cleanup.sh` stops all services, prunes Docker resources, removes build artifacts — full environment reset
– `bootstrap.sh` recreates venv and validates system integrity through integration tests
– MLflow file-based tracking survives API restarts when `runtime/mlruns` is on a persistent volume

## Production Considerations

### Multi-Tenancy

Production platforms isolate teams via namespaces, API keys, and resource quotas. Each team sees only their experiments, models, and cost records. Our platform uses a single `MLOpsState` — production would shard state by `team_id`.

### Security

– Enable authentication middleware on all `/API/*` routes
– Restrict CORS to known dashboard origins
– Encrypt MLflow artifact storage (S3 SSE, GCS CMEK)
– Run containers as non-root users
– Scan model artifacts for serialized pickle vulnerabilities

### RBAC

Map roles to API capabilities:

### scalability

– Horizontal API scaling behind a load balancer (stateless control plane)
– External state store (Redis/PostgreSQL) replacing in-memory `MLOpsState`
– Async job queue (Celery, SQS, Kafka) for training and pipeline work
– CDN for static dashboard assets

### Observability

– Prometheus metrics (implemented)
– Structured JSON logging with trace IDs
– Grafana dashboards for module-level SLOs
– Alertmanager rules on drift alerts, error rates, cost anomalies

### Failure Handling

– Circuit breakers on downstream service calls
– Retry with exponential backoff on training failures
– Dead letter queues for failed pipeline stages
– Graceful degradation: dashboard shows stale metrics with timestamp when API is degraded

### Disaster Recovery

– MLflow tracking server backed by S3 with cross-region replication
– database backups for governance and audit data
– Documented runbook for full platform restore
– Regular restore drills (not just backup verification)

### Cost Optimization

– Spot/preemptible instances for fault-tolerant training
– Autoscaling inference based on request rate
– Model quantization to reduce serving infrastructure
– Cost allocation tags per team and experiment

### performance Bottlenecks

## Advanced Patterns

### Controller Pattern

Implement a Kubernetes Operator that watches `ModelDeployment` CRDs and reconciles:

– Desired model version and traffic percentage
– Actual Deployment/Service/Ingress state
– Status conditions reported back to the CRD

Our `mlops_state` SYNCERS are a microcosm of this pattern.

### Async Workflows

Replace synchronous training calls with:

1. API accepts job, returns `job_id` immediately
2. Worker picks up job from queue
3. Status updates via WebSocket or polling
4. Completion triggers governance workflow automatically

Argo Workflows and Kubeflow Pipelines implement this at cluster scale.

### Event-Driven Architecture

Publish domain events to Kafka:

– `model.training.completed`
– `pipeline.validation.failed`
– `monitoring.drift.detected`

Downstream consumers trigger retraining, alerting, or cost analysis without tight coupling.

### Multi-Cluster Strategies

– Control plane in a management cluster
– Training workloads on GPU clusters
– Inference on regional edge clusters
– GitOps (ArgoCD) syncs model deployment manifests across clusters

- Hands-On Tutorial

systemdesign02

Hands-On System Design Tutorial

## 1. Introduction

A scheduler can fire on time, but the work still has to land somewhere. In production systems, that landing zone is almost always a message broker — RabbitMQ, Kafka, or something similar — and a fleet of consumer processes that pull execution requests off a queue and run them. When consumers are too slow, queues back up and SLA windows slip. When they acknowledge messages too early, work is lost on crash. When they never acknowledge, the same message replays forever. When you add instances without understanding prefetch and fair dispatch, one consumer hogs the backlog while others sit idle.

The `week7_integrated_project` (under `week_7_part1_taskscheduler_integrated_project/`) is a single Spring Boot application that walks through three successive layers of consumer engineering across Days 33–35. Day 33 establishes the baseline: consume `Day33TaskExecutionRequest` messages from RabbitMQ, process them asynchronously, persist lifecycle state in H2, and publish status updates. Day 34 shifts the reliability conversation to Kafka — manual offset acknowledgment, typed failure outcomes, exponential-backoff retries, and a dead-letter topic. Day 35 closes the arc with horizontal scaling: multiple consumer threads (and optionally multiple JVM instances) competing on a shared `task.queue`, with Redis tracking per-instance throughput.

This article is written for backend engineers who already understand REST APIs and basic messaging and want to see how consumer correctness and scale are implemented in real Spring Boot code — not as abstract patterns, but as named classes you can step through in a debugger.

—

## 2. From Fundamentals to a Unified System

### Day 33 — Consuming Task Execution Requests from a Message Queue

Day 33 answers the first question every consumer team faces: *how do I pull work off a broker and actually execute it without blocking the listener thread?* The entry point is `Day33TaskConsumer`, annotated with `@RabbitListener` on the `task-execution-queue` (configured in `Day33RabbitConfig` with 3–10 concurrent consumers via `day33RabbitListenerContainerFactory`). When a `Day33TaskExecutionRequest` arrives, the listener delegates to `Day33TaskProcessor.processTask()`, which is marked `@Async` and runs on virtual threads (`spring.threads.virtual.enabled: true` in `application.yml`).

State is not ephemeral. `Day33TaskProcessor` loads or creates a `Day33Task` entity and persists transitions through `Day33TaskRepository` — `QUEUED` → `PROCESSING` → `COMPLETED` or `FAILED`. After each transition, `Day33StatusPublisher` emits a JSON status update to the `task-status-exchange` topic exchange. Publishers enter through `Day33TaskPublisherService`, which is also exposed via `Day33ApiController` at `POST /API/day33/tasks`.

In production, the failure mode this solves is the “fire-and-forget consumer” — a service that logs a message and returns without tracking whether execution actually succeeded. When the process dies mid-task, operations has no record of what was in flight. JPA-backed `Day33Task` rows give you an auditable execution ledger tied to a `workerId` from `InstanceIdProvider`.

### Day 34 — Acknowledging Messages and Handling Failures in Consumers

Day 34 tackles the harder problem: *when is it safe to tell the broker I’m done?* Kafka auto-commit is convenient and dangerous — a crash after processing but before commit replays the message; a commit before processing loses it. `Day34KafkaConfig` disables auto-commit (`ENABLE_AUTO_COMMIT_CONFIG: false`) and sets `AckMode.MANUAL_IMMEDIATE` on `day34KafkaListenerContainerFactory`. `Day34KafkaTaskConsumerService` listens on `task-execution` and `task-retry` topics, deserializes JSON into `Day34Task`, and routes to pluggable processors (`Day34EmailTaskProcessor`, `Day34ReportTaskProcessor`) via the `Day34TaskProcessor` interface.

Each processor returns a `Day34ProcessingResult` with status `SUCCESS`, `RETRYABLE_FAILURE`, or `PERMANENT_FAILURE`. `Day34MessageAcknowledgmentHandler` is the decision engine: success calls `acknowledgment.acknowledge()` immediately; retryable failures with retries remaining invoke `Day34RetryHandler.scheduleRetry()` (exponential backoff capped at 60 seconds) and then ack the original offset; exhausted retries or permanent failures route through `Day34DeadLetterHandler` to the `task-dead-letter` topic before acking. `Day34ConsumerMetrics` records counters and processing timers for the dashboard at `/day34/dashboard`.

The production failure mode here is the **infinite redelivery loop** — a poison message that crashes the consumer on every attempt, wedging the partition. Explicit acknowledgment only after a defined outcome, plus DLQ routing, breaks that loop while preserving the message for inspection.

### Day 35 — Scaling Task Consumers Horizontally

Day 35 asks: *given a growing backlog, how do I add capacity without breaking fairness or losing observability?* `Day35TaskProducerService` publishes `Day35ScalingTask` messages to the durable `task.queue` (declared in `Day35RabbitConfig`). `Day35ScalingConsumerService` competes for messages with a listener factory tuned for scaling: `prefetch-count: 1` and `max-concurrent-consumers: 3` prevent one thread from hoarding messages. Each consumer instance identifies itself via `week7.day35.consumer-instance-name` (overridable with `CONSUMER_ID`), and writes per-instance counters to Redis keys like `stats:{consumerId}:processed`.

`Day35MetricsService` aggregates Redis stats and queries RabbitMQ queue depth through `day35RabbitAdmin.getQueueInfo()` for the dashboard at `/day35/dashboard`. You can simulate true horizontal scaling by launching a second JVM on port 8081 with `CONSUMER_ID=consumer-2`.

The production failure mode is **false scaling** — running three processes on one machine but configuring prefetch so high that only one ever receives work, or scaling out without per-instance metrics so you cannot tell which consumer is lagging.

Together, the three days form a progression inside one JVM. Day 33 proves you can consume and track work. Day 34 proves you can consume *correctly* under failure. Day 35 proves you can consume *at scale* with observable distribution. They share `Week7ModuleProperties` for feature toggles, `JacksonConfig` for JSON serialization, and `Week7IntegratedApplication` as the single entry point — but intentionally use separate brokers queues/topics so each lesson’s boundaries remain visible in package names (`day33`, `day34`, `day35`).

—

## 3. Architecture Overview

The system has five conceptual layers, all hosted inside one Spring Boot process unless you explicitly launch additional instances for Day 35 scaling experiments.

**Entry and routing layer.** `Week7IntegratedApplication` bootstraps the JVM with `@EnableAsync`, `@EnableKafka`, and `@EnableRetry`. `RootController` serves the landing page at `/`, and each day’s `Day33DashboardController`, `Day34DashboardController`, and `Day35DashboardController` expose module-specific dashboards and REST APIs.

**Configuration layer.** `Week7ModuleProperties` centralizes per-day settings (`week7.day33.*`, `week7.day34.*`, `week7.day35.*`). Broker-specific beans live in `Day33RabbitConfig`, `Day34KafkaConfig`, `Day35RabbitConfig`, and `Day35RedisConfig`. `SharedRabbitConfig` provides the shared `Jackson2JsonMessageConverter`.

**Consumer processing layer.** Day 33: `Day33TaskConsumer` → `Day33TaskProcessor` → type-specific execution methods. Day 34: `Day34KafkaTaskConsumerService` → `Day34TaskProcessor` implementations → `Day34MessageAcknowledgmentHandler`. Day 35: `Day35ScalingConsumerService` with simulated work via `Thread.sleep(task.getComplexityMs())`.

**Persistence and observability layer.** Day 33 uses H2 via JPA (`Day33Task` / `Day33TaskRepository`). Day 34 uses in-memory `Day34TaskHistoryService` plus Micrometer via `Day34ConsumerMetrics`. Day 35 uses Redis (`Day35MetricsService`, `Day35RedisConfig`).

**External broker boundary.** RabbitMQ (Day 33 execution queue + status exchange; Day 35 work queue), Kafka (Day 34 execution/retry/DLQ topics), Redis (Day 35 metrics), and H2 (Day 33 task ledger) sit outside the JVM and are started via `docker/docker-compose.yml`.

Dependency flow is inward: controllers and listeners call services; services call repositories, Redis templates, or broker templates; all broker connections flow through Spring Boot auto-configuration pointed at `localhost` by default.

—

## 4. Data / Control Flow

The primary unit of work across this project is a **task execution message** — a broker-delivered payload describing work to perform. The richest control-flow path is Day 34’s Kafka pipeline, because it encodes the acknowledgment decision that Days 33 and 35 simplify or defer.

**Entry.** A task is published to the `task-execution` topic by `Day34TaskProducerService` (triggered via `POST /day34/API/tasks/email` or `/day34/API/tasks/batch`). The message is a JSON-serialized `Day34Task` with fields `id`, `type`, `payload`, `retryCount`, and `maxRetries`.

**Consumption.** `Day34KafkaTaskConsumerService.consumeTask()` receives the raw string, deserializes it, and selects a `Day34TaskProcessor` where `canProcess(task.getType())` returns true. The processor executes simulated I/O (email send, report generation) and returns a `Day34ProcessingResult`.

**Decision.** `Day34MessageAcknowledgmentHandler.handleProcessingResult()` branches on `result.getStatus()`:

– **SUCCESS** → `acknowledgment.acknowledge()`, increment success metric, record processing time.
– **RETRYABLE_FAILURE** with `task.hasRetriesLeft()` → `Day34RetryHandler.scheduleRetry()` publishes to `task-retry` after `2^retryCount` seconds (plus jitter), then acks the current offset.
– **RETRYABLE_FAILURE** with exhausted retries, or **PERMANENT_FAILURE** → `Day34DeadLetterHandler.sendToDeadLetterQueue()` publishes to `task-dead-letter`, then acks.

**Cyclical outcome.** Retried tasks re-enter through the same listener (both `task-execution` and `task-retry` topics are subscribed). **Terminal outcomes** are a successfully acked offset (work done) or a record on the dead-letter topic (work abandoned with audit trail).

Day 33 follows a simpler path: `Day33TaskPublisherService` → `task-execution-queue` → `Day33TaskConsumer` → async `Day33TaskProcessor` → H2 status update + `Day33StatusPublisher`. Day 35 follows: `Day35TaskProducerService` → `task.queue` → `Day35ScalingConsumerService` → Redis status keys.

—

## 5. State Management

The most completely modeled execution state in the integrated project is the `Day33Task` JPA entity (`@Table(name = “day33_tasks”)`). It tracks the lifecycle of a single execution request from the broker through to completion.

**Fields that constitute state:**

**Lifecycle transitions:**

1. `Day33TaskPublisherService.publish()` creates a row with `status = QUEUED` before the message hits RabbitMQ.
2. `Day33TaskProcessor.processTask()` sets `PROCESSING`, assigns `workerId`, records `startTime`.
3. On success → `COMPLETED` + `completedTime`.
4. On exception → `FAILED`, `errorMessage` set, `retryCount` incremented.

Day 35 mirrors a lighter version in Redis: `task:{id}:status` cycles through `QUEUED` (set by producer), `PROCESSING`, `COMPLETED` or `FAILED`. Day 34 does not persist task rows; instead `Day34ProcessingResult.Status` represents the *message handling outcome* for a single delivery attempt, while `Day34Task.retryCount` travels with the message across retry publications.

The state machine diagram below focuses on `Day33Task.TaskStatus` because it is the persisted, queryable execution ledger exposed on `/day33/dashboard` and `/API/day33/stats`.

—

## 6. Key Engineering Insights

**Acknowledge after outcome, not after receipt.** Day 34’s `MANUAL_IMMEDIATE` ack mode means the offset is not committed until `Day34MessageAcknowledgmentHandler` explicitly calls `acknowledge()` — and that call happens only after the handler knows whether to retry, dead-letter, or complete. This is the difference between at-least-once processing with controlled redelivery and silent message loss.

**Separate concurrency from correctness.** Day 33 scales listener threads (`min-concurrent-consumers: 3`, `max-concurrent-consumers: 10`) while Day 34 scales Kafka container concurrency (`consumer-concurrency: 3`). Both increase throughput, but neither substitutes for acknowledgment discipline. You can run ten threads and still lose messages if you ack too early.

**Prefetch governs fair horizontal distribution.** Day 35 sets `prefetch-count: 1` in `Day35RabbitConfig`’s listener factory so no single consumer thread buffers a large batch of unacked messages. Without this, adding instances does not reliably increase throughput — one greedy consumer starves the rest.

**Persist execution state outside the message.** Day 33’s `Day33Task` entity survives broker redelivery and gives the dashboard queryable history. Day 35’s Redis keys serve a similar observability role at higher volume. Relying solely on in-flight message state makes crash recovery a guessing game.

—

## 7. Success Metrics

### Modularity
– [ ] Each day’s code lives in its own package (`day33`, `day34`, `day35`) with no cross-day service imports.
– [ ] `Week7ModuleProperties` toggles days independently via `week7.day33.enabled`, `week7.day34.enabled`, `week7.day35.enabled`.
– [ ] Broker configs are isolated in `Day33RabbitConfig`, `Day34KafkaConfig`, `Day35RabbitConfig` with `@Qualifier`-disambiguated templates.

### Readability
– [ ] Listener entry points are one class per day: `Day33TaskConsumer`, `Day34KafkaTaskConsumerService`, `Day35ScalingConsumerService`.
– [ ] Day 34 failure routing is centralized in `Day34MessageAcknowledgmentHandler`, not scattered across processors.
– [ ] Dashboard routes follow a consistent pattern: `/day33/dashboard`, `/day34/dashboard`, `/day35/dashboard`.

### Correctness
– [ ] Day 34 disables Kafka auto-commit and uses manual ack in `Day34KafkaConfig`.
– [ ] Retryable failures increment `retryCount` on the `Day34Task` message before republication to `task-retry`.
– [ ] Day 33 `Day33Task` transitions are persisted before status is published via `Day33StatusPublisher`.

### Extensibility
– [ ] New Day 34 task types require only a new `Day34TaskProcessor` implementation with `canProcess()` — no listener changes.
– [ ] Day 35 consumer identity is externalized to `CONSUMER_ID` / `week7.day35.consumer-instance-name` for multi-JVM scaling.
– [ ] Day 33 task types extend via the `switch` in `Day33TaskProcessor.executeTaskByType()` without touching `Day33TaskConsumer`.

—

## 8. Conclusion

The underlying principle Week 7 teaches is **consumer responsibility**: a task consumer is not merely a message handler — it is a stateful worker that must decide when work is truly complete, how to behave under failure, and how to scale without hiding bottlenecks. The integrated project makes that responsibility concrete across RabbitMQ and Kafka, with JPA and Redis providing the observability layer operations teams actually need.

From here, a natural next step is consumer group rebalancing under partition migration (Kafka), dead-letter replay tooling, and idempotent processing guards — ensuring that the at-least-once delivery guarantees you have carefully built do not corrupt downstream state when messages arrive more than once.

- Hands-On Tutorial

systemdr5

Hands-On System Design tutorial with practical examples and real-world applications.