Less 9: High Availability and Fault Tolerance

High Availability and Fault Tolerance for Log Ingestion

Summary

• A Spring Boot log platform (platform-ha-ft) composes failover, multi-region replication, circuit breakers, backpressure, and chaos testing in one ingest pipeline

• Profile toggles (local, kafka, docker, redis) switch between H2/in-memory stores and ZooKeeper, Kafka, Redis, Postgres, and Prometheus

• Events are gated, fenced by leadership, deduplicated, reordered, persisted behind Resilience4j breakers, then published to Kafka or an in-memory queue

• Operators verify behavior via dashboard demos, Micrometer metrics, and a chaos suite that injects failures and checks recovery

Core Concepts

High availability for log ingestion is a pipeline of controls, not a single hot-standby process. This system places four protective layers on the write path before an event is considered accepted:

1. Backpressure — shed load early with a rate limiter and lag-aware throttling

2. Active-passive failover — only the leader processes; standbys buffer without split-brain writes

3. Multi-region replication — Bloom + exact dedup and a watermark reorder buffer keep peer-region copies coherent

4. Circuit-guarded persistence — Postgres and Redis breakers degrade to a bounded write buffer instead of failing the request path

Chaos experiments close the loop: validate steady state, inject failure, measure resilience, recover, and re-validate. That workflow mirrors how mature SRE teams prove availability claims under controlled stress.

Context in Distributed Systems