Lesson 3 : Distributed Locking

Lesson 3 — Mutual Exclusion at Fleet Scale: Distributed Locks for Task Schedulers

Why This Matters

You already have a catalog of schedules and a binder that turns rows into ScheduledFutures. That is a control plane. Horizontal scale breaks it the moment two JVMs share the same catalog: each node still wakes on the same cron, and both happily “reconcile invoices” at 02:00. Payment platforms call that double-settlement. Warehouse systems call it double-ship. Schedulers call it the reason locks exist.

Airflow’s executors, Kubernetes leader election for controllers, and Redis-backed queue claim patterns all solve variants of the same question: among N healthy processes, which one may enter this critical section right now? integrated lab — Aegis — makes that question tangible. You will watch unprotected ticks collide, then enforce exclusion with database leases, optimistic claims, Redis SET NX, and a Redlock quorum — one dashboard, no toy mocks.

The production insight that separates junior from senior designs is not “use Redis.” It is knowing which failure mode your lock actually covers. A Postgres SELECT FOR UPDATE and a Redis key with TTL fail differently under network partitions. Product managers feel those differences as duplicate emails; SREs feel them as pager storms; architects must name them before choosing a store.

Agenda

1. The single-instance illusion after scale-out

2. Where locks sit in the Task Scheduler architecture

3. Control flow: tick → acquire → run → release

4. Data flow across RDBMS and Redis

5. Lock and task state machines

6. Four real coordination strategies (plus the “none” baseline)

7. Build, test, and demo Aegis

8. Production trade-offs and homework

The Problem Being Solved

Static @Scheduled methods are process-local. Persist the schedule in a database and every replica will still fire. Without a distributed lock, your audit table fills with two EXECUTED rows for the same logical tick — different instanceId, same business effect.

That is not a logging bug. It is a missing mutual-exclusion boundary around work that is not idempotent. Even idempotent handlers benefit from locks: you save CPU, cache stampedes, and load on downstream banks or SaaS APIs that rate-limit by account.

Why It Matters in Distributed Systems

At ultra-scale you rarely run one scheduler pod. You run many for availability. Availability without exclusion multiplies side effects. Classic distributed systems literature frames this as safety vs liveness: locks protect safety (“at most one critical section”), while leases and TTLs protect liveness (“a dead holder cannot block forever”).

Systems students already recognize include:

• Stripe / payment ledgers — settlement jobs must not post twice.

• GitHub Actions / CI schedulers — cron workflows across runners need claim semantics.

• Kubernetes — lease objects elect a controller leader so reconcile loops do not fight.

• Sidekiq / queue workers — BRPOP and unique locks stop duplicate job execution.

Aegis does not reinvent ZooKeeper. It teaches the interfaces those systems share: acquire with timeout, hold with expiry, release only if you still own the token.

Fit in the Overall Task Scheduler