Day 16: Logging & Tracing

System Design with Data Structures & Algorithms ~7 min read 60 min hands-on

Alright team, welcome back. Today, we're diving into the eyes and ears of any robust distributed system: Logging and Tracing. You might think, "Logs? Traces? I've seen them before." But trust me, at the scale we're talking about – handling 100 million requests per second – these aren't just debug tools; they are the central nervous system that keeps your complex organism alive and observable. Without them, you're flying blind in a storm.

The Unseen Battle: Why Observability is King at Scale

Imagine you're running a global e-commerce platform. A customer complains about a failed payment. In a monolith, you might check a single log file. In a distributed system with microservices, queues, databases, and third-party APIs, that single request might traverse dozens of services. Where did it fail? Was it the payment gateway? A database timeout? A network glitch between two services? Pinpointing the issue without proper observability is like finding a needle in a haystack... on fire... in the dark.

This is where Logging and Tracing step in, not as separate entities, but as complementary forces.

Core Concepts: Logging – The System's Diary

Logging is the practice of recording discrete events or messages generated by your application or system components. Think of it as each service maintaining its own diary.

  1. Structured Logging: Forget plain text. At scale, logs must be structured (e.g., JSON). Why? Because you need to query them programmatically. A log line like INFO: User 123 failed payment is useless. A structured log like {"level": "INFO", "timestamp": "...", "service": "payment-service", "event": "payment_failed", "user_id": "123", "transaction_id": "abc-123"} is a treasure trove. You can filter by user_id, service, event, and quickly find patterns.

  2. Log Levels: DEBUG, INFO, WARN, ERROR, CRITICAL. These aren't just suggestions; they're critical for controlling log volume. In production, you typically run at INFO or WARN to prevent your log pipeline from drowning in DEBUG messages. But be ready to dynamically switch to DEBUG for specific services during incident response.

  3. Log Aggregation: Individual service logs are fragmented. You need a centralized system (like ELK Stack: Elasticsearch, Logstash, Kibana, or Loki/Prometheus/Grafana) to collect, store, and visualize all logs from all services. This is non-negotiable for distributed systems.

  4. Cost & Retention: Logs are data, and data costs money (storage, ingress/egress, processing). At 100M RPS, log volume can be astronomical. Develop a clear strategy for retention (e.g., 7 days for detailed logs, 30 days for error logs, 90 days for audit logs) and consider intelligent sampling or redaction for less critical data.

Core Concepts: Tracing – The Request's Journey

While logs tell you what happened in a specific service, tracing tells you how a single request flowed through all services. It's the story of one request's entire journey.

Keep going with the full lesson

The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.

Already a member? Sign in