Day 1: Write the Protobuf Contract for Inventory Allocation — and Assert Wire-Format Backward Compatibility

Day 1: Write the Protobuf Contract for Inventory Allocation — and Assert Wire-Format Backward Compatibility

The Production Stakes: The Cost of a Reassigned Tag

In 2018, a major financial platform suffered a partial outage when a developer cleaned up a seemingly unused field in an internal gRPC service contract. The field deprecated_user_priority at tag 3 was removed, and a new field transaction_retry_count (an integer) was assigned to tag 3 in the same message.

When the updated service was deployed, older clients still running in the cluster continued to send payloads containing the original priority string at tag 3. The newly deployed servers parsed this incoming string data as an integer. Because Protobuf does not transmit field names—only field numbers and basic wire types—the server interpreted the raw string bytes as a varint. This caused silent data corruption, arithmetic overflows, and a cascade of service panics that took four hours to trace.

This is the reality of hyperscale microservices. When you serve millions of requests per second, you cannot orchestrate a "big bang" release where every client and server upgrades simultaneously. Your API contracts must remain backward-compatible across multiple versions.

Today, we begin building our high-performance Inventory Allocation Engine. In this lesson, you will write its core interface contract using Protocol Buffers (Protobuf) and build an automated defense pipeline using buf to prove that no developer can commit a wire-format breaking change to your repository.

The Core Concept: Wire-Format vs. Human Format

To understand why Protobuf is highly performant—and why it is fragile if mismanaged—we must look at how it serializes data compared to JSON.

Consider a simple JSON payload:

{"order_id": "ord-99", "quantity": 5}

In JSON, every message is self-describing. The wire format includes the field names ("order_id", "quantity") as ASCII text. This is highly flexible but incredibly wasteful. This tiny payload consumes 37 bytes.

Now look at how Protobuf encodes the equivalent message. It compiles the schema down to a compact binary stream. The field names are completely discarded. Instead, the wire format uses a Tag-Length-Value (TLV) layout:

Each field is serialized as a header (containing the field number and its wire type) followed by the payload bytes.

The Math of Varints

Protobuf uses Varints (variable-length integers) to save space. A standard 32-bit integer normally takes 4 bytes. In Protobuf, small integers take only 1 byte.

The most significant bit (MSB) of each byte in a varint is a continuation bit. If it is set to 1, more bytes are coming; if 0, this is the last byte. The remaining 7 bits store the actual two's complement representation of the number in least-significant-group first order.

For example, the integer 5 fits in one byte: 00000101 (MSB is 0).
The field header is calculated as:
$$text{Header} = (text{fieldnumber} ll 3) mid text{wiretype}$$

For our quantity field (let's say it is field number 2, wire type 0 for varint):
$$text{Header} = (2 ll 3) mid 0 = 16 = text{0x10 in hex}$$

The entire serialized payload for quantity = 5 is just 2 bytes: 10 05.

This efficiency is why Protobuf payloads are typically 80% to 90% smaller than JSON. However, this also means the field number (tag) is the sole identifier of your data. If you change the data type of tag 2, or assign a different field to tag 2, the parsing engine will decode the binary stream into the wrong fields without throwing a validation error.

Component Architecture

Our system consists of three main components:

1. 1.The API Contract (inventory.proto): Defines the RPC methods and binary message structures for our Inventory Allocation service.

2. 2.The Linter and Breaking-Change Guard (buf): A modern toolchain that enforces style guides and asserts wire-format compatibility against previous versions of our schema.

3. 3.The Microservice Loop (Go gRPC Server & Client): A high-performance execution loop that implements the contract, serving allocations with zero-copy serialization.

Design Trade-offs: Protobuf vs. FlatBuffers

While Protobuf is our choice for internal RPCs, it is not the only binary format. FlatBuffers is a common alternative.

• Protobuf wins when memory footprint is a priority. It unpacks the wire format into memory-allocated structs, making it easy to manipulate programmatically.

• FlatBuffers wins in ultra-low latency scenarios (like game development or high-frequency trading) because it allows access to serialized data without a deserialization step (zero-copy memory mapping). However, FlatBuffers schemas are more complex to manage and generate larger wire payloads.

Code Walkthrough: Designing for Resiliency

Below is a critical slice of our inventory.proto schema. Notice how we explicitly manage field numbers and use enums with an explicit UNSPECIFIED zero-value.

syntax = "proto3";

package api.v1;

option go_package = "github.com/resilient-api/inventory-allocation/gen/go/api/v1;apiv1";

// Best Practice: Always define an explicit UNSPECIFIED zero-value for enums.
// In proto3, the default value for any enum is the 0th element. If a client
// fails to set a value, it defaults to 0. An explicit UNSPECIFIED value
// prevents business logic from accidentally choosing a valid state.
enum AllocationFailureReason {
  ALLOCATION_FAILURE_REASON_UNSPECIFIED = 0;
  ALLOCATION_FAILURE_REASON_OUT_OF_STOCK = 1;
  ALLOCATION_FAILURE_REASON_INVALID_SKU = 2;
  ALLOCATION_FAILURE_REASON_WAREHOUSE_CLOSED = 3;
}

message AllocationItem {
  string sku = 1;         // Unique identifier for the stock keeping unit
  uint32 quantity = 2;    // Must be greater than 0
  string warehouse_id = 3; // Source fulfillment center
}

By enforcing a style where every enum has an UNSPECIFIED value at tag 0, we prevent bugs where a missing field in a client request is interpreted as a valid business state (such as OUT_OF_STOCK).

The Failure Demo: Breaking the Wire Contract

To truly understand how our defenses work, we will deliberately attempt to introduce a breaking change to our API contract.

In the implementation phase, you will write a clean version of inventory.proto. You will then simulate a developer trying to optimize the schema by changing the type of the order_id field from a string to an int32 on tag 1 to save space.

When you run the compatibility suite, the buf breaking tool will analyze the abstract syntax tree (AST) of both schemas, detect that tag 1 changed from wire type 2 (length-delimited string) to wire type 0 (varint), and abort the build before any code is compiled or deployed.

Production Realities and Laptop Simplifications

In a true hyperscale production environment:

• Schema Registries are integrated into the CI/CD pipeline. Every commit to a proto file triggers a check against the production schema registry (such as Buf Schema Registry or Confluent Schema Registry) to prevent breaking changes from ever merging to the main branch.

• Backward compatibility checks are run against the exact version currently serving live traffic in production, not just the local file system.

On your laptop, we simplify this workflow by saving a snapshot of our valid schema to a local directory (compat_check/) and running our breaking-change assertions directly against that snapshot. This gives you the exact same safety guarantee without the overhead of hosting a remote schema registry.

Assignment: Protect the Warehouse Allocation Schema

Your task is to extend the inventory.proto contract to support multi-warehouse fallback routing, while maintaining strict backward compatibility.

Success Criteria

1. 1.Add a new message type FallbackWarehouse containing a string warehouse_id (tag 1) and a uint32 priority (tag 2).

2. 2.Add a repeated field of type FallbackWarehouse named fallback_warehouses to the AllocateRequest message.

3. 3.Ensure that running buf breaking --against compat_check passes successfully.

4. 4.Intentionally reassign the tag number of fallback_warehouses to 1 (colliding with order_id), run buf breaking, and verify that the tool successfully catches and flags the collision.

Hints

• Never reuse an existing tag number within the same message, even if you delete the old field. If you must delete a field, use the reserved keyword (e.g., reserved 4;) to prevent future developers from claiming that tag.

• The field type repeated changes how data is packed on the wire. Changing a field from optional/single to repeated is a breaking change for existing deserializers.

In our next lesson, Day 2: The Tension, we will dive deeper into how client applications crash when these rules are violated, and build a resilient parsing engine that gracefully handles schema drift.