Day 1: The 8GB Budget: Why Your Kubernetes Tutorial Is Lying To You

Implementation Deep Dive: Enforcing the Budget

Step 1: Understanding Linux Memory Management

Before we configure anything, you need to understand how Linux will kill your processes.

When physical RAM is exhausted:

1. Kernel tries to free page cache

2. If that fails, activates OOM killer

3. OOM killer scores processes (high memory + low OOM score adjustment = first to die)

4. Kills highest-scoring process

Docker containers have no protection by default. If your K3d cluster consumes 6GB and you open Chrome, the kernel might kill kube-apiserver instead of Chrome tabs.

We prevent this with cgroups v2 memory limits:

# Docker daemon with hard memory ceiling
dockerd --memory 4g --memory-swap 4g

The --memory-swap 4g is critical: it sets swap limit equal to memory limit, meaning no swap usage. Swap on an 8GB machine is a death sentence (thrashing).

Step 2: K3d Configuration for Survival

Standard K3d install:

k3d cluster create # Uses ~1.2GB

Our install:

k3d cluster create nano-substrate 
  --servers 1 
  --agents 0 
  --k3s-arg "--disable=traefik@server:0" 
  --k3s-arg "--disable=metrics-server@server:0" 
  --k3s-arg "--kube-apiserver-arg=--max-requests-inflight=200@server:0" 
  --k3s-arg "--kube-apiserver-arg=--max-mutating-requests-inflight=100@server:0" 
  --k3s-arg "--etcd-arg=--quota-backend-bytes=2147483648@server:0"

Line-by-line analysis:

• --servers 1 --agents 0: Single-node cluster. No "HA" pretenses.

• --disable=traefik: Traefik consumes 200MB. We'll use Cilium's Ingress Controller.

• --disable=metrics-server: We'll use kubectl top with custom metrics later.

• --max-requests-inflight=200: Throttle API server concurrency (default: 400). Prevents request queue memory explosion.

• --max-mutating-requests-inflight=100: Mutating requests (POST/PUT/PATCH) consume more memory. Cut in half.

• --quota-backend-bytes=2147483648: Limit etcd database to 2GB. Default is 8GB (absurd for our use case).

The result: K3d cluster using 600-700MB under normal load.

Step 3: Measuring the Reality

After cluster creation, verify with:

# Container memory usage
docker stats --no-stream k3d-nano-substrate-server-0

# Breakdown by Kubernetes components
kubectl top pods -n kube-system

Expected output:

NAMESPACE     NAME                                      CPU(cores)   MEMORY(bytes)
kube-system   coredns-xxx                               3m           28Mi
kube-system   local-path-provisioner-xxx                1m           18Mi

Total: ~600MB for the cluster. Compare this to a standard K3s install at 1.2GB. We just saved 600MB - enough to run 5 FastAPI services.

The 8GB Constraint: System-Level Tunings

Docker Daemon Configuration

Edit /etc/docker/daemon.json:

{
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 64000,
      "Soft": 64000
    }
  },
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "storage-driver": "overlay2",
  "default-address-pools": [
    {
      "base": "172.17.0.0/16",
      "size": 24
    }
  ]
}

Critical settings:

• log-opts: Prevents Docker logs from consuming GB of disk (and buffer cache).

• storage-driver: overlay2: Modern, memory-efficient.

Restart Docker with memory limit:

sudo systemctl stop docker
sudo dockerd --memory 4g --memory-swap 4g &

Kernel Tuning

Edit /etc/sysctl.conf:

# Reduce swap tendency
vm.swappiness=10

# OOM killer tuning
vm.overcommit_memory=1
vm.panic_on_oom=0

Apply:

sudo sysctl -p

Rationale:

• vm.swappiness=10: Kernel will avoid swap unless critically needed.

• vm.overcommit_memory=1: Allow memory overcommit (processes can allocate more than physical RAM). Necessary for Kubernetes.

• vm.panic_on_oom=0: Don't panic and reboot on OOM - let the OOM killer handle it.

Step-by-Step Lab

Prerequisites

• Ubuntu 22.04 or Fedora 38+ (8GB RAM, 40GB disk)

• Docker 24.0+ installed

• kubectl installed

• No swap enabled (sudo swapoff -a)

Execution

Run the generated setup_lesson_01.sh:

chmod +x setup_lesson_01.sh
./setup_lesson_01.sh

This script will:

1. Configure Docker with memory limits

2. Apply kernel tunings

3. Create K3d cluster with aggressive resource constraints

4. Generate a monitoring dashboard (monitor.sh)

Verification

Check memory usage:

# System-wide
free -h

# Docker ceiling enforcement
docker info | grep Memory

# K3d cluster
kubectl top nodes
kubectl top pods -A

Expected results:

• Docker shows 4GB memory limit

• K3d node shows ~600-700MB used

• Total system memory usage: ~2.5-3GB (Docker + OS)

Run the monitoring script:

./monitor.sh

This will show real-time memory allocation every 5 seconds. Leave it running and try deploying a simple pod:

kubectl run nginx --image=nginx:alpine --limits="memory=64Mi,cpu=100m"

Watch the monitor - you should see a small spike (~64MB), then stabilization.

Homework: Day 2 Optimization Challenge

Task: Reduce K3d cluster memory footprint below 500MB.

Hints:

1. Investigate --kube-apiserver-arg=--feature-gates=.... Can you disable unused features?

2. CoreDNS runs with default memory limits. Can you tune it further?

3. Research K3s's --snapshotter option. Can you use a lighter backend?

Deliverable: Submit your modified k3d cluster create command and kubectl top output showing <500MB usage.

Stretch Goal: Configure Docker to use cgroups v2 unified hierarchy for even tighter memory control.

Conclusion: The Foundation Matters

Most platform engineers inherit clusters with 8-16GB per node. When you build on constraints, you develop instincts:

• "Do I really need Prometheus, or can I use Grafana Loki with a 1-day retention?"

• "Can this StatefulSet actually be stateless with proper external state management?"

• "Does this CRD need a webhook, or can we use validation rules in the schema?"

These questions lead to better architectures. An IDP that runs on 8GB will scale to 1000 nodes more efficiently than one that needs 16GB per control plane.