Lesson 5: Production Operations

Production Operations Control Plane — Operating Twitter at 10M Concurrent Users

Hands-on Twitter System Design Course — Build Production-Ready Social Media Architecture From Scratch

What We Build Today

You will assemble a unified operations control plane: one dashboard and one API that coordinate observability, incident response, GitOps rollouts, disaster recovery, auto-scaling, security operations, performance budgets, FinOps, data pipelines, SRE on-call workflows, production readiness scoring, and documentation handoff. At hyperscale, the product is only as reliable as the control surface operators trust when traffic spikes.

Today’s agenda

• Why a single ops plane beats twelve disconnected tools

• Domain architecture and request flow

• Step-by-step build, test, and demo

• Assignment: extend one domain with a new SLO signal

Core Concepts

Control plane vs data plane
The data plane serves tweets and timelines. The control plane answers: Are we within SLO? Should we fail over? Can we afford this scale-out? Netflix’s Spinnaker and Google’s SRE tooling follow the same split—product code stays fast; operators get a dedicated surface for decisions.

Domain isolation with shared transport
Each operations domain owns a service module (ObservabilityService, AutoscalingService, …) mounted under /api/*. Shared Express middleware handles metrics, health, and WebSocket fanout. This mirrors how large platforms attach new operational capabilities without rewriting the gateway.

Mutable demo contract
Every workflow endpoint returns counters that change on each call—SLO compliance drift, replica counts, servers managed, running pipelines. The UI binds fetchedAt so operators (and you, while learning) can prove the system is live, not a static mock.

Context in Distributed Systems