Day 2: Persist Service State and Externalize Configuration – Surviving a Restart

In our previous lesson, we containerized a minimal service, treating it as an isolated, ephemeral process. While isolation is a virtue, it creates a harsh reality: when a container stops, everything it "remembered"—its logs, its database files, its configuration tweaks—vanishes into the ether. Today, we move beyond the ephemeral, learning how to anchor state to the host and inject configuration from the outside, ensuring our service survives the inevitable lifecycle of a container.

The Persistence Problem

In distributed systems, the "ephemeral filesystem" is a feature, not a bug. It forces you to treat your application as a stateless entity that can be destroyed and recreated at will. However, services need to store state—like the write-ahead logs (WAL) in a database or local cache files—to function.

Consider the etcd project, a distributed key-value store used as the backbone of Kubernetes. When an etcd node restarts, it must reconstruct its state from a persistent storage directory. If that directory were trapped inside the container's volatile writable layer, a simple docker rm would result in total data loss. In production, we solve this by mounting a host directory into the container, a pattern known as a Bind Mount.

Architecture: Decoupling State and Code

To make our service resilient, we must separate the "What" (the code) from the "How" (the configuration and state).

1. 1.Configuration Injection: Instead of hardcoding settings into the image, we pass them via Environment Variables. This allows the same image to behave differently in staging versus production.

2. 2.State Persistence: We map a specific path inside the container to a directory on the host. This ensures that even if the container process dies, the data remains safely on the disk, ready to be picked up by the next instance.

The Configuration Pattern

We avoid hardcoded files by reading the environment. This is a standard practice in systems like Redis or PostgreSQL, where configuration is injected to tune performance based on the specific hardware of the host.

# Conceptual snippet: Injecting configuration
import os

# Instead of: host = "localhost"
# We use:
db_host = os.getenv("DB_HOST", "default-db-host")

The Persistence Mapping

When we map a volume, we are essentially creating a "wormhole" between the container’s filesystem and your laptop. The container believes it is writing to /data, but those bits are physically landing on your SSD.

# Conceptual snippet: Mapping the volume
# -v [host_path]:[container_path]
docker run -v $(pwd)/data:/app/data my-service

The Failure Demo: The Vanishing Act

To understand why this matters, we will simulate a "crash-and-recover" event.

1. 1.We will write data to our service.

2. 2.We will kill and rm the container.

3. 3.We will start a new container pointing to the same host directory.

If we haven't implemented persistence, the second container will see an empty directory. With persistence, the data will be there, waiting. This is the exact mechanism that prevents total loss when a node in a cluster like Kafka or Elasticsearch is replaced.

Real-World Stakes

In 2012, a misconfiguration in a major cloud provider led to the loss of data for several persistent volumes because the storage mapping was decoupled from the compute instance's lifecycle. When the compute instance was terminated during an automated scaling event, the "persistent" mapping was lost because it wasn't explicitly anchored to a persistent block device. You are learning the foundation of that anchoring today.

Moving Forward

By decoupling state, we’ve made our service "restartable." In Day 3, we will investigate the "So What" of this architecture: how docker build creates the layered images that hold your code, why those layers are often inefficient, and how to optimize them to ensure your CI/CD pipeline doesn't stall due to bloated build times.

Assignment

Your task is to modify the provided service to store its "hit count" in a file on the disk rather than in memory.

1. 1.Run the service with a bind mount.

2. 2.Increment the counter.

3. 3.Terminate the container.

4. 4.Run a new container pointing to the same host folder.

5. 5.Verify the counter reflects the previous state.

Success Criteria:

• The counter persists across container recreations.

• The start.sh script produces a final log output confirming the state was recovered.

Solution

Hint: Use a simple text file inside the /data directory. Ensure your container has write permissions to that directory. If you see "Permission Denied," remember that the container runs as a specific user (usually root) and needs to own the directory it writes to on your host.

Implementation Guide

1. 1.Prerequisites: Ensure Docker is installed and the day2-persistence repository is cloned locally.

2. 2.Setup: Navigate to the day2-persistence directory.

3. 3.Run: Execute ./start.sh. This script will:

• Build the image.

• Create a local data/ directory.

• Run the container with a bind mount.

• Simulate a restart by killing the container and starting a new one.

1. 1.Verification:

• The terminal will display the final count.

• Expected Output: Successfully recovered state: 5

1. 1.Cleanup: Run ./stop.sh to remove the containers and the local data/ directory.

#!/bin/bash
set -e
echo "Building image..."
docker build -t persistence-demo .
mkdir -p data
echo "Starting first container..."
docker run -d --name service-1 -v $(pwd)/data:/app/data persistence-demo
echo "Simulating workload (writing state)..."
docker exec service-1 sh -c "echo 5 > /app/data/counter.txt"
echo "Killing container..."
docker rm -f service-1
echo "Starting second container (recovery)..."
docker run --rm -v $(pwd)/data:/app/data persistence-demo cat /app/data/counter.txt
echo "Verification complete."