Lesson 6: Schema Evolution, Serialization & Log Compaction

Hands On Kafka Course ~5 min read 60 min hands-on

What We Will Build

  • A unified Schema Platform for StreamSocial: JSON ingress validation, Avro encoding, Protobuf vs Avro benchmarks, compatibility governance, and compacted user-preference topics.

  • A metrics dashboard where each capability renders as its own live cardοΏ½not a raw JSON dump.

  • Production-shaped patterns you can wire to Confluent Schema Registry and real Kafka topics without redesigning the architecture.

Where This Fits in StreamSocial

Reliable producers and consumers move bytes through Kafka. The harder problem at scale is contract safety: ten teams shipping schemas, mobile clients on different revisions, and JSON payloads that explode bandwidth across regions.

LinkedInοΏ½s registry discipline, UberοΏ½s schema CI gates, and WhatsApp-style settings stored on compacted topics all share one insight: the log is only trustworthy when the contract on the wire is governed.

Core Concepts

JSON Schema Is the Gate, Not the Transport

Validation catches bad emails, missing identifiers, and malformed interaction payloads before they reach consumers. At social-scale volume, one unchecked field can crash an entire consumer fleet.

Avro Shrinks the Wire; Registry Coordinates Readers

Binary layout plus schema identity in the message header lets consumers decode without embedding full schemas. You trade CPU on encode/decode for cross-AZ egress savingsοΏ½capacity planning must include both.

Protobuf vs Avro Is a Client and Platform Choice

Mobile teams often prefer Protobuf codegen; data platforms often standardize on Avro + Schema Registry. Measure bytes and nanoseconds on your payload shapeοΏ½blog benchmarks rarely match your nested fields.

Keep going with the full lesson

The rest of this lesson β€” plus its implementation guide, runnable scripts and source code β€” is included with the course or an All Access plan. Create a free account to start the free lessons in every course.

Already a member? Sign in