Lesson 5 : Stateful Workloads

Stateful Forge: Production Stateful Workloads on Kubernetes

What We Build Today

• Stateful Forge Platform — a multi-tier data platform with PostgreSQL transactions, Kafka streaming, CSI-backed event ingestion, and Velero-ready backup orchestration

• Tiered persistence layer — StorageClasses, StatefulSets, PVCs, and snapshot classes wired for production storage semantics

• Operations control plane — RBAC deployer, storage monitor, backup controller, and a unified API gateway

• Failure injection lab — debug scenarios for pending PVCs, quota exhaustion, and cross-namespace RBAC misconfigurations

Why This Matters

Netflix, Spotify, and Airbnb run thousands of StatefulSets alongside Deployments because state is not an implementation detail—it is a contract with your customers. When a pod restart loses transaction history or a PVC binds to the wrong zone, the blast radius exceeds any single microservice outage. Stateful workloads introduce ordering constraints, identity requirements, and backup dependencies that Deployments never surface. Platform teams that treat databases and message brokers as "just another pod" consistently underestimate recovery time. The gap between a demo Postgres container and a production-grade stateful tier is measured in RBAC boundaries, storage class selection, backup RPO/RTO, and the ability to diagnose Pending PVCs under pressure. This week closes that gap with one integrated system you can deploy, break, and restore on a local cluster.

Kubernetes Architecture Deep Dive

StatefulSet Identity vs Deployment Elasticity

StatefulSets assign stable network identity (pod-0, pod-1) and bind PVCs to pod ordinal—a pattern PostgreSQL replication and Kafka brokers depend on. Deployments excel at horizontal scale with interchangeable replicas; StatefulSets excel when pod name and storage affinity matter. Trade-off: StatefulSet rollouts are slower and ordered; you cannot freely scale down without orphaning PVCs. Anti-pattern: Running PostgreSQL as a Deployment with a single RWO PVC—rescheduling after node loss may fail because the volume cannot attach to a differently named pod. Insight: Choose StatefulSets when identity and storage coupling are part of your availability model, not when you only need persistence for one replica.