Lesson 1: Event-Driven Architecture & Kafka Foundations

Event-Driven Architecture and Kafka Foundations

Introduction

Most Kafka tutorials stop at kafka-console-producer. Engineers learn topics, offsets, and consumer groups in isolation, then discover in production that reliability, partitioning, and failure handling were never composed into a coherent system.

Distributed event-driven platforms introduce partial failures across brokers, skewed partition load, duplicate delivery, and observability gaps that surface only under real throughput. StreamSocial is a production-oriented reference platform: each subsystem adds a concrete operational concern to a running deployment, not an abstract exercise.

StreamSocial models a social engagement pipeline—likes, shares, comments, and system signals—flowing through a multi-broker Kafka cluster, into consumers, and out to metrics and storage. The platform establishes event taxonomy, cluster topology, partition strategy, high-volume producers, and resilient consumers with retry and dead-letter handling.

Real-time architectures matter because user-facing products no longer tolerate batch-only analytics. Feeds, notifications, fraud checks, and ML feature pipelines depend on events arriving in order, at scale, with clear failure boundaries. StreamSocial provides a working environment to reason about those boundaries before adding stream processing, public APIs, or extended observability.

System Overview

StreamSocial is an integrated event platform: Docker-hosted Kafka (KRaft, three brokers), Python producers and consumers, Redis-backed caching, and a FastAPI operations dashboard. Events originate from simulated user activity, publish to category-specific topics, and are consumed by an engagement pipeline that updates metrics and handles errors.

Components

Event propagation

• Taxonomy validation — Ten core event types map to three categories and three topics (src/events/).

• Topic provisioning — TopicManager creates topics with configured partitions and replication factor 3.

• Publish — Producers serialize JSON events, route by category, and key partitions for ordering where required.

• Consume — EngagementConsumer polls content-interactions, deserializes, and runs engagement logic.

• Observe — Metrics, cluster health, and runtime state surface on the dashboard API.

Architecture Diagram

The operations UI drives the runtime controller, which orchestrates producers and consumers against the broker tier. Storage sits adjacent to the processing path for cache and DLQ persistence.

Core Concepts

Event-Driven Architecture

What it does. Components communicate by publishing facts (events) instead of synchronous RPC chains. Producers remain unaware of downstream consumers.

Where it is used. src/events/models.py defines EventType, categories, and topic routing. EventPublisher emits domain events into Kafka.

Why it matters. Decoupling allows analytics, moderation, and notification consumers without redeploying producers.

Apache Kafka and Distributed Logs

What it does. Kafka stores ordered, replicated partitions across brokers. Producers append; consumers track offsets.

Where it is used. docker/docker-compose.yml runs a three-broker KRaft cluster on ports 9092–9094. config/kafka_config.py centralizes broker and client settings.

Why it matters. The log is the system of record. APIs, caches, and dashboards are projections of that log.

Partitioning

What it does. Topics split into partitions for parallelism. Keys route related events to the same partition for per-key ordering.

Where it is used. src/partitioning/partition_calculator.py models 50M req/s sizing; topic_manager.py provisions user-actions (12 local / 1000 high-scale), content-interactions (6 / 500), and system-events (3).