Welcome to Day 1 of our "254-Day Distributed Log Processing System Implementation" journey! Today, we'll establish the foundation for your distributed system by configuring a professional development environment.

What Are Distributed Systems?

Imagine organizing a large school event. If one person handled everything—decorations, food, music, and tickets—they'd quickly become overwhelmed. Instead, you divide responsibilities among team members who coordinate through group chats. That's the essence of a distributed system: multiple computers working together as one cohesive unit, sharing information to solve problems too large for any single machine.

Why Docker, Git, and VS Code?

Before constructing a skyscraper, you need the right tools and a solid foundation. For our distributed system:

• Docker creates consistent environments across different computers (similar to providing identical science lab kits for every student)

• Git tracks changes and enables seamless collaboration (like maintaining organized notes that everyone can access and modify)

• VS Code provides a powerful, extensible editor optimized for our workflow (like having a fully customizable workbench)

Setting Up Your Environment

Let's install and configure each tool systematically:

1. Install Git

Git is our version control system—it tracks code changes and facilitates collaboration.

Windows:

• Download Git from git-scm.com

• During installation, accept default options

Mac:

• Open Terminal and execute: xcode-select --install

• Follow prompts to install developer tools (includes Git)

Linux:

• Open Terminal and execute: sudo apt-get update && sudo apt-get install git (Ubuntu/Debian)

• For other distributions, use your package manager

Verify installation by opening a terminal/command prompt and typing:

git --version

2. Install Docker

Docker enables us to create containerized environments that function identically across any system.

Windows/Mac:

• Download Docker Desktop from docker.com

• Install and complete the setup wizard

• Docker Desktop includes Docker Engine, Docker CLI, Docker Compose, and essential tools

Linux:

• Follow the official Docker installation guide for your distribution

Verify installation:

docker --version
docker-compose --version

3. Install VS Code

Visual Studio Code will serve as our integrated development environment (IDE).

• Download from code.visualstudio.com

• Install using default settings

4. Install Essential VS Code Extensions

Open VS Code and install these extensions (click the Extensions icon in the sidebar):

• Docker

• Remote - Containers

• Python

• GitHub Pull Requests and Issues

• YAML

Creating Your Project Repository

Now let's create and structure your project:

1. Open Terminal/Command Prompt

2. Create your project directory:

mkdir distributed-log-processor
cd distributed-log-processor

3. Initialize Git repository:

git init

4. Create your project structure:

mkdir -p src/services config data docs tests
touch README.md docker-compose.yml .gitignore

5. Create a comprehensive .gitignore file:

echo "# Python" > .gitignore
echo "__pycache__/" >> .gitignore
echo "*.py[cod]" >> .gitignore
echo "*$py.class" >> .gitignore
echo ".env" >> .gitignore
echo "venv/" >> .gitignore
echo "data/" >> .gitignore

6. Create your first Docker Compose file:

docker-compose.yml

version: '3'

services:
  # Simple Python service to test the environment
  logger:
    build:
      context: ./src/services/logger
    volumes:
      - ./src/services/logger:/app
    ports:
      - "8000: 8000"
    environment:
      - LOG_LEVEL=INFO

7. Create your first Python service:

src/services/logger/app.py

import os
import time
from datetime import datetime

def main():
    """Simple logger service to verify our environment is working."""
    print("Starting distributed logger service...")
    log_level = os.environ.get("LOG_LEVEL", "INFO")
    print(f"Log level set to: {log_level}")
    
    while True:
        timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
        print(f"[{timestamp}] Logger service is running. This will be part of our distributed system!")
        time.sleep(5)

if __name__ == "__main__":
    main()

src/services/logger/Dockerfile

FROM python:3.9-slim

WORKDIR /app

# Copy requirements first to leverage Docker cache
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY . .

# Run the application
CMD ["python", "app.py"]

src/services/logger/requirements.txt

# No external dependencies for our simple service yet
# We'll add more as we expand our distributed system

README.md

# Distributed Log Processing System

This project is part of the "254-Day Distributed Log Processing System Implementation" series.

## Project Overview 

We're building a distributed system for processing log data at scale. This repository contains all code and configuration needed to run the system.

## Getting Started

### Prerequisites
- Docker and Docker Compose
- Git
- VS Code (recommended)

### Running the Application
1. Clone this repository
2. Navigate to the project directory
3. Run `docker-compose up`

## Project Structure
- `src/services/`: Individual microservices
- `config/`: Configuration files
- `data/`: Data storage (gitignored)
- `docs/`: Documentation
- `tests/`: Test suites

## Day 1 Milestones
- Development environment setup
- Project structure creation
- Basic logger service implementation

Architecture Diagram

[Diagram placeholder]

Data Flow Diagram

[Diagram placeholder]

Build It Today: Your First Distributed System Component

Now let's bring everything together and test your setup! Follow these steps to see your logger service in action:

1. Open Terminal/Command Prompt and navigate to your project directory:

cd distributed-log-processor

2. Build and start your Docker container:

docker-compose up --build

If everything is configured correctly, you should see your logger service start and print timestamped messages every 5 seconds.

Congratulations! You've successfully:

• Set up a complete development environment

• Created a structured project repository

• Built a simple containerized service

The Distributed Systems Connection

You might wonder: "How is this a distributed system? It's just one service!"

Excellent observation! Today we've built the foundation that enables us to add multiple interconnected services. Think of it as constructing the first LEGO brick of what will become a massive LEGO city.

In the coming days, we'll add more services that communicate with each other to form a true distributed system. Our logger service will eventually connect to other components and process log data from multiple sources simultaneously—similar to how different departments coordinate during a major school event.

Complete Source Code

[Link to complete code]

Looking Ahead

In our next session, we'll:

• Add a service that generates log data

• Create communication channels between services

• Implement basic error handling and recovery mechanisms

Success Criteria

You've completed Day 1 successfully if:

✅ Docker, Git, and VS Code are installed and functioning
✅ You've created a project repository with the correct structure
✅ Your logger service runs in a Docker container
✅ Timestamp messages appear every 5 seconds

Real-World Connection

The environment you've just configured mirrors what professional developers use to build systems like:

• Netflix: Uses containerized services to stream videos to millions of users

• Spotify: Processes music streaming data using distributed log analysis

• Instagram: Handles billions of photos with distributed processing systems

Each of these platforms started with a foundation similar to what you've built today!

Remember: The most complex distributed systems in the world begin with a single service. You've taken your first step toward building something incredible!

Day 1 Assignment: Extending Your Distributed Logger

Objective

Enhance your basic logger service by adding configuration options and improving log output formatting.

Tasks

1. Create a Configuration File

Create a config.py file in your logger service directory with these requirements:

• Define log levels (DEBUG, INFO, WARNING, ERROR, CRITICAL)

• Add configurable log format options

• Include a setting for log frequency (replacing the hardcoded 5 seconds)

2. Implement Log File Output

Modify your logger service to:

• Write logs to both console AND a log file

• Create a volume mapping in Docker to persist logs on your host machine

• Implement log rotation (create a new log file when size exceeds 1MB)

3. Add a Simple Web Interface

Create a minimal web interface that:

• Displays the most recent logs

• Shows the current configuration

• Runs on port 8080

4. Document Your Changes

Update the README.md file to:

• Explain your new features

• Add instructions for accessing the web interface

• Document available configuration options

Submission Guidelines

• Push your changes to your Git repository

• Take a screenshot of your logger output and web interface

• Be prepared to demonstrate your solution in the next session

Extension Challenge (Optional)

If you finish early and want an extra challenge:

• Implement a second service that generates random log events

• Make your logger service receive and process these events

• Document how these two services work together

Evaluation Criteria

• Clean, well-commented code

• Proper use of Docker volumes for persistence

• Correct implementation of configuration options

• Working web interface

• Updated documentation

Solution for Homework

https://github.com/sysdr/course/tree/main/day1/homework

Files included:

• src/services/logger/config.py

• src/services/logger/logger.py

• src/services/logger/web_server.py

• src/services/logger/app.py

• docker-compose.yml (Updated)

• src/services/logger/Dockerfile (Updated)

• README.md (Updated)

63 Likes ∙ 13 Restacks

Discussion about this post

Write a comment...

Nemesis Prime · 5 Aug
Thanks for the series. I see the future lessons are under subscription. Is there a plan to make them available for free in future?

Liked (1) · Reply · Share
1 reply by System Design Course

Alex Chan · Alex's Substack · 15 Jul
I'm not seeing Remote - Containers in the extensions marketplace. Has this been replaced with something else?

Like (2) · Reply · Share
1 reply

2 more comments...

Related Posts

Hands On System Design with "Distributed Systems Implementation - 254-Lesson's curriculum"
254-Lesson's Distributed Log Processing System Implementation
May 13, 2025 · 140 💬 27 🔄 34

Hands On coding Course: System Design: From Zero to Production
Master System Design Through Hands-On Implementation
May 12, 2025 · 100 💬 8 🔄 16

Day 3: Building a Distributed Log Collector Service
Real-time File Watching with Event-Driven Architecture
Oct 5, 2025 · 9 💬 5

Day 8: Building a TCP Server for Network-Based Log Collection
Introduction
May 19, 2025 · 34 💬 7

Day 4: Log Parsing - Extracting Structure from Chaos
Week 1: Setting Up the Infrastructure
May 16, 2025 · 35 💬 6

Day 5: Building a Log Storage System with Rotation Policies
Welcome to Day 5 of our journey into distributed systems!
May 17, 2025 · 19 💬 5

Day 2: Production-Ready Log Generator
Building Real-World Event Processing Systems
Oct 1, 2025 · 6 💬 1 🔄 1

Day 15: JSON Support for Structured Log Processing
Transforming Raw Logs into Structured Data Gold
May 26, 2025 · 17 💬 6

Day 123: AWS CloudWatch Integration - Pulling Logs from the Cloud
What You're Building Today
Dec 1, 2025 · 11 💬 6

Day 13: TLS Encryption for Secure Log Transmission
Building Trust in Your Distributed Log Processing System
May 24, 2025 · 14 💬 4

© 2026 System Design Course · Privacy ∙ Terms ∙ Collection notice
Start your Substack
Get the app
Substack is the home for great culture