Day 28: Define Service Level Objectives (SLOs) and Configure Anomaly Detection Alerts — and Get Paged Proactively

On Day 27, we successfully instrumented DurableKV with high-resolution Prometheus metrics, exposing raw counters for requests, errors, and histogram buckets for latency. Today, we transition from merely observing our system to defending it. We will construct a production-grade alerting engine that monitors these metrics to detect anomalies before they exhaust our error budget. On Day 29, we will leverage these alert states to trigger automated rate-limiting and failover mitigations.

The Production Stakes: The Silent Budget Bleed

In September 2015, AWS experienced a severe DynamoDB outage in the US-East-1 region that cascaded into dozens of dependent services. A primary cause of the prolonged recovery was metric dilution and lack of proactive, rate-of-consumption alerting. Standard static alerts (such as "error rate > 1% for 5 minutes") either flooded operators with transient noise or failed to flag localized partition failures that were slowly but surely destroying the system's overall availability budget.

Most teams monitor systems using static thresholds:

This approach is fundamentally broken. If your system experiences a 4.9% error rate, it will never trigger this alert. Yet, at a 4.9% error rate, a 99.9% monthly availability budget (which permits only 43 minutes of total downtime per month) is completely exhausted in less than 15 hours. Conversely, a transient 100% error spike lasting 10 seconds will trigger the alert, waking up an engineer at 3 AM for a self-healing blip.

To survive at scale, we must alert on Error Budget Burn Rate using multi-window, multi-burn-rate configurations.

The Intuition: The Fuel Gauge vs. The Fuel Flow Meter

Think of your Service Level Objective (SLO) as a road trip.

• Your Error Budget is the fuel in your gas tank. If you commit to a 99.9% success rate over 30 days, your error budget is 0.1% of all requests. This is your fuel.

• A Static Alert is like a warning light that only turns on when you have 1 gallon left. If you are driving at 100 mph, you will run out of fuel in minutes. If you are idling, you could last hours.

• Burn Rate Alerting is a flow meter. It calculates how fast you are consuming your fuel relative to your remaining time.

A burn rate of 1 means you will consume exactly 100% of your error budget over your target window (e.g., 30 days). A burn rate of 14.4 means you will consume 2% of your entire monthly budget in 1 hour. If we detect a burn rate of 14.4, we must page an engineer immediately, because at this rate, we will be completely bankrupt of reliability in 36 hours.

To avoid alert fatigue, we use Multi-Window Alerting. We only fire an alert if both a short window (e.g., 5 minutes) and a long window (e.g., 1 hour) exceed the burn rate threshold. The short window ensures we respond rapidly to catastrophic spikes, while the long window ensures we do not page on transient spikes that resolve themselves within seconds.

Component Architecture

Our alerting architecture consists of three core components integrated directly alongside the DurableKV storage node: