Day 52 : The FTL Engine: High-Performance Query Logging

Welcome back, fellow architects and engineers! Today, we’re diving deep into a topic often overlooked in its foundational importance: high-performance query logging. In the sprawling landscapes of distributed systems, logs aren't just afterthought text files; they are the digital pulse of your infrastructure, the forensic trail for debugging, and the raw data for critical observability.

If you've ever found yourself staring at a production incident, wishing you had more granular, real-time insights into what queries were hitting your services, or which internal DNS lookups were failing, then you understand the pain. Mainstream logging solutions are often heavy, resource-intensive, or simply not designed for the sheer volume of query-level events that modern systems generate. Imagine a system handling 100 million requests per second; each request might trigger multiple internal queries (database, cache, DNS, internal APIs). Logging all of that efficiently is a non-trivial system design challenge.

Today, we're building the "FTL Engine" – our custom Faster Than Light query logging system. It's a lightweight, performant solution designed for the home lab, but architected with principles directly applicable to hyperscale environments. We'll focus on logging DNS queries, a common bottleneck and visibility black hole in many setups, but the core design patterns are universal.

Agenda: Navigating the Query Stream

1. The "Why" of FTL: Understanding the limitations of traditional logging for high-volume queries.

2. Core Concepts: Structured Logging, Append-Only Data, Buffering/Batching, and Log Rotation.

3. FTL Engine Architecture: Deconstructing our custom query logger and its components.

4. Control & Data Flow: Tracing how query events move through our system.

5. State Management: The lifecycle of our log buffer.

6. Real-World Application: How these concepts scale to 100M RPS.

7. Hands-On Assignment: Building the FTL Engine.

Core Concepts: Beyond the Basics

System Design Concept: Structured Logging & Append-Only Data Stores

At 100M RPS, writing plain text grep-able logs for every internal query quickly becomes a bottleneck. Disk I/O, parsing overhead, and storage costs skyrocket. The FTL Engine tackles this with two core ideas:

1. Structured Logging: Instead of [timestamp] INFO Query for example.com from 192.168.1.1, we extract key-value pairs: {"timestamp": "...", "client_ip": "...", "domain": "...", "type": "A"}. This makes logs machine-readable, enabling efficient querying and analysis without expensive regex parsing. In our home lab, we'll use JSON Lines (JSONL) format, which is easily parsable and still human-readable.

2. Append-Only Data Stores: For write-heavy workloads, appending new data to the end of a file is vastly more efficient than modifying existing records or managing complex database indexes. Our FTL Engine writes to compressed, append-only files. This minimizes disk seek times and simplifies the write path, crucial for high throughput. Think of how Kafka or typical WAL (Write-Ahead Log) systems operate.

Architecture: The FTL Engine Blueprint

Our FTL Engine consists of two primary components:

• ftld (FTL Daemon): This is the heart of our system. It's a lightweight Go program that acts as a log collector. It monitors a source log file (simulating dnsmasq output), parses incoming DNS queries, buffers them in memory, and periodically flushes these structured logs to compressed, rotated files on disk.

• ftlctl (FTL Control): A companion Go CLI tool. This is our "dashboard" for interacting with the FTL Engine. It reads the compressed log files, decompresses them on the fly, and allows us to query recent DNS activity or view aggregated statistics like top queried domains.