Lesson 3: Resource Governance, Persistent Storage, and Service Mesh Ingress
System Scope
K8 Advanced Forge is a production-oriented log analytics platform built on Kubernetes. The stack combines FastAPI microservices, a React operations console, a Kafka ingestion pipeline, and PostgreSQL on StatefulSets with persistent volumes. The platform integrates resource governance, durable storage, edge ingress, Istio service mesh security, and weighted canary routing across multiple API gateway revisions.
Core deliverables:
Resource governance — ResourceQuota, LimitRange, Guaranteed and Burstable QoS tiers, HPA on the CPU-intensive processor tier, VPA recommendations on analytics, PodDisruptionBudgets on critical paths.
Durable data plane — StatefulSets for PostgreSQL and Kafka with bound PVCs, stable network identity, and retained volume lifecycle independent of pod rescheduling.
North-south traffic control — NGINX Ingress for path-based routing and rate limits; Istio Gateway for mesh-native ingress with mTLS, AuthorizationPolicies, and weighted traffic splits (70/25/5) across three API gateway revisions.
Failure injection workspace — isolated
forge-debugnamespace with intentional DNS and connectivity faults for multi-tier incident response practice.
Operational Context
At scale, Kubernetes functions as a capacity allocator and traffic governor, not merely a deployment runtime. Organizations operating thousands of services depend on honest resource requests to prevent silent node overcommit. Unscoped ingress endpoints can amplify traffic unexpectedly. Resource requests are scheduling contracts; PVCs are storage contracts; Ingress and Istio configurations are contracts with external and internal clients. Treating any of these as afterthoughts produces costly production incidents during peak load or regional failures.
Architecture Deep Dive
Resource Requests as Scheduling Contracts
The scheduler places pods using requests, not limits. A Burstable pod with a 200m CPU request and a 2 CPU limit can be densely packed until burst traffic starves neighboring workloads — the classic noisy-neighbor failure mode. Log-ingestion receives Guaranteed QoS (requests equal limits) because it anchors the critical ingest path. Log-processor retains Burstable limits to absorb parsing spikes. HPA on log-processor scales on CPU and memory utilization only when requests reflect actual demand.
Anti-pattern: setting requests to 1m to maximize schedulability — HPA and the scheduler lose visibility into real utilization.
Design principle: limits protect the node; requests protect the SLO.
StatefulSets and Storage Binding
Deployments are replaceable; StatefulSets carry identity. Kafka and PostgreSQL require stable DNS (postgres-0.postgres) and PVCs that survive pod eviction. The platform uses a local-storage StorageClass with pre-provisioned PVs for kind-based development. Production deployments should use CSI drivers (EBS, PD) with WaitForFirstConsumer binding.
Anti-pattern: running PostgreSQL as a Deployment with emptyDir — data loss on eviction is guaranteed.
Design principle: persistence is a property of the volume lifecycle, not the pod lifecycle.
Multi-Tier Connectivity Diagnostics
When a three-tier application fails, symptoms mislead. A frontend CrashLoop often indicates backend DNS mismatch (api-backend versus debug-backend-api), not a defective container image. NetworkPolicies introduce an additional failure dimension: pods appear healthy while traffic is dropped at L4. The forge-debug workspace enforces systematic triage: inspect endpoints, validate DNS resolution, compare Service selectors with Pod labels, then evaluate NetworkPolicy egress rules.
Design principle: in Kubernetes, the network is an API — operate it with the same rigor as application APIs.
Ingress Controllers versus Service Mesh Gateways
NGINX Ingress terminates HTTP at the cluster edge with annotations for rate limiting, CORS, and body size — sufficient for many production deployments. Istio Gateway combined with VirtualService enables L7 routing inside the mesh, retries, timeouts, and subset-based load balancing without reloading controller configurations. The trade-off is operational surface area: Ingress is a single controller; Istio adds sidecars, a control plane, and mTLS certificate rotation. Large platforms often adopt Ingress for public entry and mesh for internal east-west trust. K8 Advanced Forge supports both deployment modes through Kustomize overlays.
Design principle: Ingress solves north-south entry; the mesh solves east-west trust and observability.
Canary Traffic as Control Theory
Blue/green deployments swap risk in bulk; canaries measure risk incrementally. Istio DestinationRule subsets (stable, canary, experimental) combined with weighted VirtualService routes send a small percentage of traffic to a new build while monitoring error rate and latency. Outlier detection ejects unhealthy endpoints before operators intervene.
Anti-pattern: routing by URL path per version — clients leak version semantics; weight-based routing remains transparent.
Design principle: canary deployments are feedback loops; traffic weights are the control variable.
Deployment Topology
Platform bootstrap — the k8-advanced-forge-platform directory contains services, Kustomize bases, Istio manifests, and the forge-debug failure injection workspace.
Persistent storage overlay — applies StorageClass, PVs, PostgreSQL and Kafka StatefulSets. Application pods may remain unready until PVCs bind; verify with kubectl get pvc.
Resource governance — namespace-scoped ResourceQuota caps aggregate CPU and memory; LimitRange prevents a single container from consuming the entire quota. HPA and PDB resources depend on a healthy metrics-server.
Ingress routing overlay — MODE=ingress with the ingress Kustomize overlay installs NGINX and routes /api to ingestion and / to the operations console.
Service mesh overlay — installs the Istio demo profile, applies Gateway and VirtualService resources, PeerAuthentication in STRICT mode, and AuthorizationPolicies. Sidecars inject via the namespace label.
Canary routing overlay — maintains three api-gateway Deployments; VirtualService splits traffic 70/25/5. The operations console reads X-Gateway-Version response headers to visualize distribution.
Validation — ingest logs, query statistics, execute load-test.sh, confirm HPA scales log-processor under synthetic load.
Production Considerations
Node drains: PDB
minAvailablemust align with HPAminReplicas— misalignment blocks voluntary disruptions indefinitely.Storage: migrate from hostPath PVs to cloud CSI; enable snapshots before schema migrations.
Mesh upgrades: pin Istio versions; validate control plane changes before rolling the data plane across clusters.
Failure recovery: monitor Kafka consumer lag during processor scale-up; watch for PostgreSQL connection pool exhaustion at HPA maximum replica count.
Enterprise Scale Patterns
Netflix Titus and Spotify's Kubernetes platforms enforce quota at namespace boundaries for team isolation, operate stateful tiers on StatefulSets, and execute progressive delivery through mesh-level traffic splitting. K8 Advanced Forge implements the same primitives — resource contracts, persistent identity, edge routing, mesh security — at a scale suitable for local kind clusters and portable to multi-region production environments.