Lesson 4 : GitOps & Declarative Operations

Lesson 4: Unified GitOps Control Plane

What We're Building Today

One platform that answers five incident questions without opening five tools: Is Git in sync? Are secrets and policy clean? Is infra provisioned? Can we restore? Are we burning SLO budget?

Agenda

• FastAPI control plane with seven domain APIs and shared PlatformState

• Operations dashboard — sync, canary, drift scan, policy audit, Crossplane claims, Velero DR, live SLO feed

• Consolidated k8s/ bundle: Argo CD, ApplicationSet, secrets, Gatekeeper, Crossplane XRD, Velero

• Runtime tooling, integration tests, and Docker Compose for local and container runs

Build Target

Ship a single declarative operations surface that composes GitOps delivery, secrets posture, policy enforcement, infrastructure claims, disaster recovery, and observability into one API and one dashboard. The cluster is not the source of truth—Git is. This build trains the handoffs where production GitOps actually breaks: policy blocking sync, drift hiding manual changes, backup windows missing compliance, sync latency rising before user errors spike.

Component Architecture

Place in the Overall System

This sits between CI and the cluster—after code merges, before pods run. Spotify’s delivery stack uses a similar split: CI builds artifacts; GitOps controllers reconcile; policy and secrets gates sit in the path; observability watches reconciliation health, not just HTTP 200s.

Product teams care because promotion delays here block features. Security cares because drift and open violations are audit findings waiting to happen. SRE cares because sync P99 is a leading indicator—Argo repo-server overload often shows up there before customer-facing error rates move.

Core Concepts

Platform Engineering

You are building the internal control plane platform teams expose to product engineering—not a demo console. One API shape per concern (sync, audit, provision) so downstream automation (runbooks, chatbots, CI gates) can call the same endpoints operators use manually.

Release Engineering

Git is the release artifact; the cluster is the deployment receipt. ApplicationSets fan out environments; Rollouts advance canary weight. The non-obvious bit: promotion is a graph, not a button—staging health, analysis score, and policy compliance must align before prod moves.

DevOps

Close the loop from commit to reconciled state. When someone kubectl patches and skips Git, drift scanners surface the gap. Your drift domain models the same question SOC2 auditors ask: does the cluster match the repo?

SecOps

Sealed Secrets stay encrypted in Git; External Secrets pull from vault at runtime. Gatekeeper denies bad manifests at admission—cheaper than evicting pods that never should have started. High-severity open violations should block sync (stretch goal below).

SRE

Error budget and sync_latency_p99_ms are leading SLIs. Shopify-scale GitOps fleets have seen incidents where deployments looked green while reconciliation backlog grew for hours. Watch controller health, not just app health.

Cloud Engineering