Day 2: Implement Concurrent Request Handling with Go Goroutines and Channels — and Observe Race Conditions

Day 2: Implement Concurrent Request Handling with Go Goroutines and Channels — and Observe Race Conditions

Welcome back, future distributed systems architects! Yesterday, we built a basic HTTP key-value service. It could PUT and GET data, but it only held values in memory, and critically, it could only handle one request at a time effectively. Start sending it many requests at once, and you'd quickly find it stumbling.

Today, we're tackling one of the most fundamental challenges in any multi-threaded or concurrent system: race conditions. You'll learn how Go's goroutines and channels enable powerful concurrency, but also how easy it is to introduce subtle bugs if you don't manage shared state carefully. By the end of this lesson, your DurableKV will be able to handle many requests simultaneously without corrupting its internal state, even if it still forgets everything on restart.

The Silent Killer: Why Your Day 1 Service Breaks Under Load

Recall our DurableKV service from Day 1. It stored data in a simple Go map[string]string. When an HTTP request arrived, our http.Handler would process it. Go's HTTP server is smart: for every incoming request, it launches a new goroutine – a lightweight thread of execution – to handle it. This is fantastic for throughput, as many requests can be processed "at the same time."

But here's the catch: what happens if two goroutines try to write to the same key in our map simultaneously? Or one tries to read while another is writing? Go's map type is not safe for concurrent use. Accessing it from multiple goroutines without synchronization will lead to undefined behavior: crashes, corrupted data, or inconsistent reads. This isn't theoretical; this is how many subtle, hard-to-debug production outages begin.

Consider the "lost update" problem. Imagine our KV store is tracking a counter for a critical service. Two concurrent requests arrive:

1. 1.Request A reads counter_key (value: "10").

2. 2.Request B reads counter_key (value: "10").

3. 3.Request A increments its local copy to "11" and writes "11" back to the map.

4. 4.Request B increments its local copy to "11" and writes "11" back to the map.

The expected final value should be "12", but it's "11". One update was lost. This scenario, where the outcome depends on the non-deterministic timing of multiple operations, is a race condition.

This isn't just a Go problem. Many real-world systems, from database transaction managers to distributed caches like Redis, employ sophisticated mechanisms to avoid race conditions. Without local protection, even a single-node system is vulnerable.

The Traffic Cop Analogy: Introducing Mutexes

Imagine our in-memory map as a busy intersection. Many cars (goroutines) want to pass through (access the map). If they all go at once, chaos ensues. A traffic cop (a mutex, short for mutual exclusion) is needed. The cop allows only one car to enter the intersection at a time. When that car is done, it signals the cop, who then allows the next car.

<img src="diagram_1_architecture.svg" alt="Component Architecture Diagram" style="width:100%; max-width:600px; box-shadow: 0 4px 8px rgba(0,0,0,0.1);">
<br/>
<em>Diagram 1: Component Architecture - Clients interact with the HTTP server, which uses a protected KV store.</em>

In Go, we use sync.Mutex to act as our traffic cop. It has two primary methods: Lock() and Unlock().

• When a goroutine calls Lock(), it tries to acquire the lock. If no other goroutine holds the lock, it acquires it and proceeds.

• If another goroutine already holds the lock, the calling goroutine blocks (pauses) until the lock is released.

• Once a goroutine is done accessing the shared resource (our map), it must call Unlock() to release the lock, allowing other waiting goroutines to proceed.

For our DurableKV service, we'll embed a sync.Mutex directly into our KV store struct and acquire/release it around any operations that modify or read the internal map.

Building a Concurrent-Safe KV Store

Let's modify our DurableKV struct. We'll add a sync.Mutex field.

// kvstore/store.go
package kvstore

import (
	"fmt"
	"sync"
)

// DurableKV represents our in-memory key-value store.
type DurableKV struct {
	data  map[string]string
	mu    sync.Mutex // The traffic cop for our map
}

// NewDurableKV creates a new instance of DurableKV.
func NewDurableKV() *DurableKV {
	return &DurableKV{
		data: make(map[string]string),
	}
}

// Put sets a key-value pair in the store.
func (kv *DurableKV) Put(key, value string) error {
	kv.mu.Lock()         // Acquire the lock
	defer kv.mu.Unlock() // Ensure the lock is released when Put exits
	kv.data[key] = value
	fmt.Printf("PUT: %s = %sn", key, value) // For demo visibility
	return nil
}

// Get retrieves a value by key from the store.
func (kv *DurableKV) Get(key string) (string, bool, error) {
	kv.mu.Lock()         // Acquire the lock
	defer kv.mu.Unlock() // Ensure the lock is released when Get exits
	value, ok := kv.data[key]
	fmt.Printf("GET: %s = %s (found: %t)n", key, value, ok) // For demo visibility
	return value, ok, nil
}

<img src="diagram_2_flowchart.svg" alt="Concurrency Flowchart" style="width:100%; max-width:600px; box-shadow: 0 4px 8px rgba(0,0,0,0.1);">
<br/>
<em>Diagram 2: Concurrency Flowchart - How requests are serialized by the mutex.</em>

Notice the defer kv.mu.Unlock() line. This is a crucial Go idiom. defer ensures that Unlock() is called just before the Put or Get function returns, regardless of how it exits (success, error, panic). This prevents common bugs like forgetting to release a lock, which would lead to a deadlock – where all subsequent goroutines trying to acquire the lock would block forever.

Trade-off: Using sync.Mutex is simple and effective for protecting small, shared data structures. However, it serializes all access to the map. If your system has many more reads than writes, a sync.RWMutex (Read-Write Mutex) might be more efficient. An RWMutex allows multiple readers to access the resource concurrently, but only one writer at a time, and writers block readers. For our DurableKV, a simple Mutex is sufficient for now, as it correctly demonstrates the core concept.

The Race Condition Failure Demo

Let's see the race condition in action and then fix it. We'll create a simple client that tries to increment a counter in our KV store concurrently.

Before: Run the Day 1 service (without sync.Mutex). We'll send 1000 concurrent requests to increment a key. If everything worked, the final value should be "1000". Without the mutex, it will almost certainly be less.

After: We'll update the DurableKV with the sync.Mutex and re-run the same test. This time, the final value should be "1000", demonstrating that our mutex correctly protected the shared state.

<img src="diagram_3_state_machine.svg" alt="Mutex State Machine" style="width:100%; max-width:600px; box-shadow: 0 4px 8px rgba(0,0,0,0.1);">
<br/>
<em>Diagram 3: Mutex State Machine - The mutex transitions between Locked and Unlocked states, arbitrating access.</em>

This is what it looks like at production scale: systems like Amazon DynamoDB or Google Spanner deal with concurrency not just within a single process but across many machines. They use more complex distributed consensus algorithms (like Raft or Paxos, or variations like Google Spanner's TrueTime) to ensure that even with many nodes processing requests, updates are applied correctly and consistently. Our local sync.Mutex is the simplest form of this, protecting a single node's state.

What We've Built, and What's Next

Before today, our DurableKV could not reliably handle concurrent requests, leading to data corruption and inconsistent reads. After today, it can safely handle many concurrent PUT and GET operations to its in-memory store. You can run curl http://localhost:8080/get/mykey after a concurrent test to prove the final value is correct.

However, our DurableKV is still just an in-memory cache. If you stop the service, all your data vanishes. This is a critical problem for any "durable" store. In Day 3, we'll tackle this head-on by implementing a Write-Ahead Log (WAL) to persist data to disk – and we'll learn how easy it is to corrupt that data if we're not careful.

Assignment: Add Concurrent-Safe DELETE

Your assignment is to extend the DurableKV service to include a DELETE /key endpoint. This endpoint should remove a key-value pair from the store. Crucially, it must also be concurrent-safe, meaning it uses the sync.Mutex (or sync.RWMutex if you're feeling adventurous) to protect access to the map during deletion.

Steps:

1. 1.Add a Delete method to the DurableKV struct in kvstore/store.go.

2. 2.Implement the Delete method to acquire the lock, remove the key from kv.data, and then release the lock.

3. 3.Add a new HTTP handler function for DELETE /key in main.go.

4. 4.Register this handler with your HTTP server.

5. 5.Modify your concurrent test (or create a new one) to include concurrent DELETE operations, interleaved with PUT and GET, to verify its correctness. For example, concurrently PUT a key, DELETE it, and GET it, ensuring the final GET correctly reports "not found".

Success Criteria:

• You can successfully DELETE a key using curl -X DELETE http://localhost:8080/delete/mykey.

• A concurrent test involving DELETE operations passes without data corruption or unexpected behavior.

Solution Hints

• The Delete method signature in kvstore/store.go could look like func (kv *DurableKV) Delete(key string) error.

• Inside Delete, remember the kv.mu.Lock() and defer kv.mu.Unlock() pattern.

• To remove an item from a Go map, use the delete built-in function: delete(kv.data, key).

• For the HTTP handler, you'll need to extract the key from the URL path, similar to your GET handler. The http.MethodDelete constant can be used for routing.

• When testing, ensure your concurrent test specifically targets a scenario where multiple DELETEs or interleaved PUT/DELETEs could cause a race if unprotected. For example, launch 100 goroutines: half PUT the same key, half DELETE it, then check the final state.