Day 3: Scale consumers without chaos

One fulfillment-service instance cannot keep up on a busy day. The obvious fix is to run three. Kafka makes that easy — every instance joins the same consumer group and the brokers split the partitions between them — and it also makes it easy to get wrong: a consumer that takes too long between polls is thrown out of the group, its partitions move, its work is redone, and the group spends its time rebalancing instead of fulfilling orders. Today you scale out, measure the gain, then cause a rebalance storm and fix it with one setting.

What you build

• Three fulfillment instances (f1, f2, f3 on ports 8082–8084) in the group fulfillment, one listener thread each, reading the 6-partition orders topic. Each event costs 10 ms of simulated work.

• RebalanceTracker, a ConsumerAwareRebalanceListener attached to the orders listener container: it counts rebalances and remembers which partitions this instance owns.

• LagService: lag per partition from the Kafka AdminClient (log-end offset minus committed offset), at GET /lag.

• Cooperative rebalancing (CooperativeStickyAssignor): when a member joins or leaves, only the partitions that have to move are revoked; the rest keep flowing.

How the group shares the work

Each partition is owned by exactly one member of a group. With 6 partitions and 3 members, each owns 2; a fourth member would get some, a seventh would sit idle. The partition count is the ceiling on parallelism for one group — choose it with growth in mind (Day 2 explains why changing it later is not free).

When a member joins, leaves or stops polling, the group coordinator (a broker) runs a rebalance and hands out partitions again. Membership is proven by polling: if a consumer doesn't call poll() within max.poll.interval.ms (here 5 s), the coordinator assumes it is stuck and evicts it.