Day 10 : Exception Handling in Distributed Systems.

Welcome back, architects! Today, we're diving into a topic often treated as an afterthought in local applications but becomes a cornerstone of resilience in hyperscale distributed systems: Exception Handling.

You've built a microservice, deployed it, and it's handling requests. Great! But what happens when things go sideways? An upstream service is down, a database connection drops, or a network hiccup occurs. In a monolithic application, an error often means a stack trace in the log and perhaps a 500-page to the user. In a distributed system handling 100 million requests per second, an unhandled or poorly handled exception isn't just an error; it's a potential catastrophe. It can lead to cascading failures, resource exhaustion, and a complete system meltdown.

This isn't about simply wrapping code in try-catch blocks. That's entry-level. This is about designing an error contract and a failure strategy that keeps your system stable and observable, even when parts of it are in turmoil.

The Blame Game: Who Owns the Error?

In a distributed system, an error rarely happens in isolation. A client calls Service A, which calls Service B, which tries to talk to a database. If the database is unreachable, where does the error originate?

• Is it Service B's fault for not handling the database error?

• Is it Service A's fault for calling Service B when it's flaky?

• Is it the client's fault for making the request at all?

This "blame game" is why a standardized approach is paramount. Every service must understand its role in failure propagation and recovery.

The Problem with Default Spring Boot Error Handling

Out-of-the-box, Spring Boot provides a basic error page (for browsers) or a generic JSON response (for APIs). While convenient for development, it often lacks:

• Consistency: Different services might return slightly different error structures.

• Context: Generic messages like "Internal Server Error" are useless for debugging or automated recovery.

• Actionability: No clear guidance for clients on how to proceed.

• Security: Sometimes, stack traces are exposed, revealing internal system details.

At 100M RPS, inconsistency means your automated retry mechanisms, monitoring dashboards, and even client applications will struggle to interpret failures correctly. This leads to manual intervention, which is impossible at scale.

Core Concept: The Error Contract – A Pact Between Services

Just as your API defines a contract for successful responses (data types, fields, etc.), it must define a contract for error responses. This is your Error Contract.

An effective error contract typically includes:

1. HTTP Status Code: The primary indicator (e.g., 400 Bad Request, 401 Unauthorized, 404 Not Found, 409 Conflict, 500 Internal Server Error, 503 Service Unavailable).

2. A Unique Error Code: A specific, domain-driven code (e.g., USER_NOT_FOUND, INSUFFICIENT_FUNDS, EXTERNAL_SERVICE_TIMEOUT). This is more precise than the HTTP status.

3. A Human-Readable Message: A concise explanation for the client (and potentially end-user).

4. Developer Message (Optional but Recommended): More technical details for engineers debugging the issue.

5. Correlation ID (Crucial): A unique identifier for the entire request chain, allowing you to trace an error through logs across multiple services.

6. Timestamp: When the error occurred.

Architecture: Global Exception Handling with @RestControllerAdvice

Spring Boot's @RestControllerAdvice (or @ControllerAdvice) combined with @ExceptionHandler is your go-to for implementing a global error contract. It centralizes exception handling, keeping your business logic clean and ensuring consistent error responses across your microservice.

How it fits: