Day 2 : OS Tuning – Why Your Kernel Is Sabotaging Your 8GB IDP

Implementation Deep Dive: The Memory Pressure API

We're building a FastAPI backend that does not use the bloated psutil library (which pulls in platform-specific C extensions and adds 15MB). Instead, we parse /proc directly.

Backend: memory_service.py

python

from pathlib import Path
from typing import Dict, Optional
from pydantic import BaseModel, Field

class MemoryStats(BaseModel):
    total_mb: int = Field(..., description="Total physical RAM")
    available_mb: int = Field(..., description="Available for allocation")
    swap_total_mb: int = Field(..., description="Total swap space")
    swap_free_mb: int = Field(..., description="Unused swap")
    swappiness: int = Field(..., description="Current kernel swappiness")
    pressure_level: str = Field(..., description="low|medium|high|critical")

def parse_meminfo() -> Dict[str, int]:
    """Parse /proc/meminfo without external dependencies."""
    meminfo = {}
    with Path("/proc/meminfo").open() as f:
        for line in f:
            if line.strip():
                key, value = line.split(":", 1)
                # Extract numeric value (strip 'kB' suffix)
                meminfo[key.strip()] = int(value.strip().split()[0])
    return meminfo

def get_swappiness() -> int:
    """Read current vm.swappiness setting."""
    return int(Path("/proc/sys/vm/swappiness").read_text().strip())

def calculate_pressure(available_mb: int, total_mb: int) -> str:
    """Determine memory pressure level."""
    ratio = available_mb / total_mb
    if ratio > 0.4:
        return "low"
    elif ratio > 0.2:
        return "medium"
    elif ratio > 0.1:
        return "high"
    else:
        return "critical"

async def get_memory_stats() -> MemoryStats:
    """Gather current memory statistics."""
    meminfo = parse_meminfo()

    total = meminfo["MemTotal"] // 1024  # Convert kB to MB
    available = meminfo["MemAvailable"] // 1024
    swap_total = meminfo.get("SwapTotal", 0) // 1024
    swap_free = meminfo.get("SwapFree", 0) // 1024

    return MemoryStats(
        total_mb=total,
        available_mb=available,
        swap_total_mb=swap_total,
        swap_free_mb=swap_free,
        swappiness=get_swappiness(),
        pressure_level=calculate_pressure(available, total)
    )

Why This Matters:

• No Dependencies: Importing psutil costs 12MB resident memory + 18MB shared libraries

• Direct Kernel Interface: /proc is the source of truth

• Async-Ready: We can stream this data at 1Hz without blocking

Frontend: Real-Time Memory Gauge

The React component uses TanStack Query to poll the API every 3 seconds (not 1s - we're not monitoring a nuclear reactor):

typescript

// hooks/useMemoryStats.ts
import { useQuery } from '@tanstack/react-query';
import { api } from '../lib/api';

export interface MemoryStats {
  total_mb: number;
  available_mb: number;
  swap_total_mb: number;
  swap_free_mb: number;
  swappiness: number;
  pressure_level: 'low' | 'medium' | 'high' | 'critical';
}

export const useMemoryStats = () => {
  return useQuery<MemoryStats>({
    queryKey: ['memory-stats'],
    queryFn: () => api.get('/memory/stats').then(r => r.data),
    refetchInterval: 3000, // 3s polling
    staleTime: 2000,
  });
};

The Constraint Enforcement:

Notice the refetchInterval: 3000. In a typical React tutorial, you'd see:

typescript

refetchInterval: 100  // BAD: 10 req/s = wasted CPU cycles

At 100ms polling:
- **CPU Cost:** ~2-5% constant background load
- **Network:** 600 requests/minute to localhost (pointless)
- **Battery Drain:** Significant on laptops

At 3000ms polling:
- **CPU Cost:** <0.5%
- **Network:** 20 requests/minute
- **User Experience:** Still feels "real-time" for memory trends

---

### The 8GB Constraint: Swappiness Math

Here's the formula that determines when Linux swaps:

swap_tendency = mapped_ratio / 2 + distress + vm.swappiness

Where:

• mapped_ratio: Percentage of RAM used by file-backed pages

• distress: How hard the kernel is trying to free memory (0-100)

• vm.swappiness: Your tunable parameter (0-100)

Default (swappiness=60):

• At 50% RAM usage, kernel starts swapping proactively

• Result: Your K3s API server gets swapped out while you have 4GB free

Nano-IDP (swappiness=10):

• Kernel only swaps when distress > 40 (i.e., genuinely struggling)

• Result: Anonymous pages stay in RAM until <1.5GB available

Setting it:

bash

Temporary (until reboot)

sudo sysctl vm.swappiness=10

# Permanent
echo "vm.swappiness=10" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Windows Equivalent:

Windows doesn't expose swappiness, but you control paging behavior via:

1. Pagefile Size: Set to 4GB fixed (not "system managed")

   *   Right-click This PC → Properties → Advanced → Performance Settings → Advanced → Virtual Memory
   *   Uncheck "Automatically manage"
   *   Set Initial = Maximum = 4096 MB

2. Memory Compression: Enable via PowerShell (Admin):

powershell

   Enable-MMAgent -MemoryCompression

• Achieves 15-20% RAM savings by compressing pages in-memory before paging

• • *

Step-by-Step Lab

Prerequisites

• K3d cluster running (from Lesson 1)

• At least 2GB free disk space for swap file

• Root/Administrator access

Execution

Part 1: Linux Swap Configuration

bash

1. Check current memory state

free -h
swapon --show
cat /proc/sys/vm/swappiness

## 2. Create optimized swap (if not exists)
if [ ! -f /swapfile ]; then
    sudo fallocate -l 4G /swapfile
    sudo chmod 600 /swapfile
    sudo mkswap /swapfile
    sudo swapon /swapfile
    echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
fi

## 3. Tune swappiness
sudo sysctl vm.swappiness=10
sudo sysctl vm.vfs_cache_pressure=50
echo "vm.swappiness=10" | sudo tee -a /etc/sysctl.conf
echo "vm.vfs_cache_pressure=50" | sudo tee -a /etc/sysctl.conf

## 4. Verify
sysctl vm.swappiness
free -h

Part 2: Deploy the Memory Monitor

bash

Clone the lesson repo (script will generate this)

cd ~/nano-idp
./setup_lesson_02.sh

## This creates:
# - backend/ (FastAPI service)
# - frontend/ (React dashboard)
# - manifests/ (K8s deployments)
# - test/ (memory pressure simulator)

## Deploy to K3d
./build_and_deploy.sh

## Verify
kubectl get pods -n nano-system
kubectl top pods -n nano-system

Part 3: Induce Memory Pressure

bash

Run the memory hog

kubectl apply -f test/memory-pressure-pod.yaml

# Watch the monitor UI
# Navigate to http://localhost:8080/memory

# Observe:
# - Available MB drops
# - Swappiness stays at 10
# - Swap usage increases ONLY when available < 1.5GB
# - No OOM kills

Verification

Success Criteria:

1. Swap is active but not thrashing:

bash

   watch -n 2 'free -h && swapon --show'

• You should see swap usage grow slowly (not instantly)

1. Pods survive memory pressure:

bash

   kubectl get pods -A | grep -v Running

• Should be empty (no OOMKilled pods)

1. API server responsive:

bash

   time kubectl get nodes

• Should complete in <1 second (even under pressure)

1. Monitor dashboard shows "medium" pressure:

   *   Open [http://localhost:8080/memory](http://localhost:8080/memory)
   *   Pressure gauge should be yellow (not red)
   

Performance Benchmark:

• • *

Homework: Day 2 Optimization

Challenge: Configure kernel memory compaction to reduce fragmentation.

Linux's buddy allocator can cause fragmentation when you churn pods. Enable proactive compaction:

bash

Enable compaction when fragmentation index > 0.5

sudo sysctl vm.compaction_proactiveness=20
echo "vm.compaction_proactiveness=20" | sudo tee -a /etc/sysctl.conf

Task:

1. Deploy 10 vClusters

2. Delete 5 random ones

3. Repeat 10 times

4. Measure fragmentation:

bash

   cat /proc/buddyinfo
   grep compact /proc/vmstat

1. Document how much memory you reclaimed via compaction

Bonus: Write a Python script that monitors /proc/buddyinfo and triggers manual compaction when fragmentation exceeds a threshold:

bash

echo 1 > /proc/sys/vm/compact_memory

• • *

Summary: The Efficiency Wins

The Philosophy:

You don't need a $128/month Datadog agent to know your memory is full. You need:

1. Kernel fundamentals

2. A 100-line Python script

3. Discipline to not install bloated observability stacks

Next lesson: Cilium eBPF Tuning – We'll replace kube-proxy with eBPF to save another 200MB and achieve 40% faster pod-to-pod latency.