Lesson 2 : Consumer Scalability, Reliability & Delivery Semantics

StreamSocial Consumer Reliability Platform

Scope

This document describes the consumer reliability platform for StreamSocial: horizontal feed processing, offset checkpointing, durable commit ordering, rebalance-safe state transfer, and idempotent billing ingestion.

Platform role

The consumer reliability layer sits between Kafka ingestion and product surfaces (feed ranking, engagement analytics, billing, notifications). Weakness here produces duplicate charges, stuck consumers, or replay storms after deploys.

Production analogues include feed fanout workers, engagement counters, payment event processors, and notification dispatch pipelines.

Consumer group scaling

Kafka assigns partitions to group members. Additional workers increase parallel throughput until partition count becomes the bottleneck.

Partition skew can limit gains even when worker count rises. Monitor per-partition lag alongside aggregate consumer count.

Operational signal: feed_scaling.active_consumers and feed_scaling.messages_processed increase during load.

Offset checkpointing

Auto-commit is convenient but weak for analytics pipelines requiring deterministic recovery. Range claiming establishes ownership before batch processing and checkpoint persistence.

Treat offsets as lease tokens with explicit commit boundaries.

Operational signal: offset_checkpoint.ranges_committed and offset_checkpoint.metrics_rows rise together.

Commit reliability