Lesson 4 : Leader Election & Reliable Execution

Lesson 4 : Leader Election & Reliable Execution in a Distributed Task Scheduler

Why This Matters

When Uber dispatches surge-pricing jobs, when Netflix refreshes CDN caches, or when your bank runs end-of-day reconciliation, exactly one scheduler instance must decide what runs and when. Run the same cron on every pod and you double-charge customers. Run none and reports never ship.

The fix is not “turn off scheduling on followers.” You need an explicit leader election contract, lease-based heartbeats, automatic failover, coordination primitives, and idempotent execution so retries never corrupt state.

Core Concepts

1. Single-Active Scheduler Pattern

In a horizontally scaled cluster, one node holds a lease in shared storage. The lease has an expiry timestamp. The leader renews it on a heartbeat interval (typically one-third of lease duration). Followers observe an unexpired lease owned by someone else and stay idle.

This is how Google’s Chubby-inspired systems, Kubernetes lease objects, and JDBC-backed Spring schedulers keep one brain in charge.

2. Database-Backed Leases

PostgreSQL stores a cluster_leadership row per service. Acquisition uses optimistic versioning plus conditional updates:

• Renew: update only if you still own a non-expired lease

• Takeover: update only if the lease is expired

• Failover: force-expire a stale leader after health checks fail

This trades a few milliseconds of latency for strong consistency without ZooKeeper.

3. Failover via Node Health