Lesson 5 : Task Lifecycle & Recovery

Lesson: Task Lifecycle & Recovery — Making Failures Observable and Survivable

Why This Matters

Gmail still delivers your mail when a SMTP relay blips. Stripe still settles payments after a network timeout. Your bank’s nocturnal batch does not “forget” mid-run because one pod restarted. They share a boring superpower: every attempt is recorded, every status change is legal, every failure is classified, and retries are bounded with backoff.

A scheduler without a lifecycle is a stopwatch with amnesia. At scale, amnesia is an outage you cannot explain.

Agenda

1. Execution ledger (who ran what, when, with what outcome)

2. Status state machine (illegal transitions rejected)

3. Error capture (message + stack persisted)

4. Durable exponential backoff (RETRYING + nextRetryAt)

5. Spring Retry for in-process transient slices

6. Dead letter when hope is gone

7. Build, test, demo

Core Concepts

1. Execution Ledger

Before you optimize throughput, you need a flight recorder. Each attempt gets a UUID executionId, start/end timestamps, duration, node id, and either an outcome or an error+stack. Netflix-style ops dashboards are useless without this row-level truth.

Insight: Metrics tell you rates; the ledger tells you stories. When a VP asks “what happened to job X at 02:14?”, you open one row—not a haystack of logs.

2. Validated Status State Machine

Statuses are not free-form strings. Treat them as a finite state machine:

PENDING → RUNNING → COMPLETED | RETRYING → RUNNING | FAILED | DEAD_LETTER | CANCELLED

Illegal hops (PENDING → COMPLETED) throw. That stops UI bugs and racey workers from inventing fantasy histories.