Building Twitter's Real-Time Notification Engine

What We're Building Today

Today we transform our static Twitter clone into a living, breathing social platform. Here's what you'll accomplish:

High-Level Agenda:

• Real-Time Tweet Delivery: Posts appear instantly in followers' timelines without page refresh

• WebSocket Connection Management: Handle 1,000 concurrent users with persistent connections

• Redis Pub/Sub Messaging: Scalable event distribution system

• Event Sourcing Architecture: Complete audit trail of all user actions

• Live Notifications System: Real-time alerts for likes, follows, and mentions

• Production-Ready Patterns: Connection recovery, heartbeat monitoring, graceful failures

Our Mission: Build a real-time event processing system that delivers tweets to 1,000 active users within 100ms, maintains complete audit trails, and handles connection drops gracefully.

Core Concepts

Real-Time Event Processing Architecture

Real-time processing is what separates social media from traditional web applications. While a blog can wait seconds to load, Twitter users expect to see new tweets the moment they're posted. This requires fundamentally different architecture patterns.

Event-Driven vs Request-Response: Traditional web apps work like a phone call - you ask a question, wait for an answer. Real-time systems work like a radio broadcast - information flows continuously to whoever's listening.

The Three Pillars of Real-Time Systems:

1. Message Broker (Redis): The highway system for events

2. WebSocket Connections: Persistent highways to each user's browser

3. Event Store: Permanent record of everything that happened

Redis Pub/Sub: The Event Highway

Redis pub/sub operates like a sophisticated radio station. Publishers broadcast messages on named channels, subscribers listen to channels they care about. Unlike traditional message queues, pub/sub delivers messages to all active subscribers simultaneously.

Why Redis for Real-Time: Redis keeps everything in memory, making message delivery microsecond-fast. It handles millions of concurrent connections and provides atomic operations crucial for consistent event ordering.

WebSocket Connections: Always-On Communication

HTTP requests are like postal letters - send a message, wait for response, connection closes. WebSockets are like phone calls - once connected, both sides can send messages anytime until someone hangs up.

Connection Management Challenge: With 1,000 concurrent users, we need to track who's online, what they're subscribed to, and handle disconnections gracefully. Users switch networks, close laptops, lose cellular signal - our system must adapt.

Event Sourcing: The System's Memory

Instead of storing current state, event sourcing stores every change as an event. Like a bank that keeps every transaction rather than just account balances. This provides complete audit trails and enables powerful analytics.

Event vs State Storage: Traditional systems store "User has 150 followers." Event sourcing stores "UserFollowedEvent at timestamp X, UserUnfollowedEvent at timestamp Y." Current state gets computed from events.

Context in Distributed Systems

Twitter's Real-Time Architecture

Twitter processes 6,000 tweets per second during peak usage. Their real-time system must:

• Deliver tweets to millions of timelines instantly

• Handle trending topic calculations in real-time

• Maintain notification systems for mentions, likes, retweets

• Provide live metrics for advertisers

Our Simplified Version: We're building the core patterns Twitter uses, scaled for 1,000 users. Same architectural principles, manageable complexity.

Integration with Previous Lessons

Our timeline generation system (Lesson 3) created the infrastructure for delivering personalized feeds. Now we make those feeds live. When someone tweets, instead of waiting for followers to refresh their pages, we push that tweet immediately to everyone's timeline.

Data Flow Integration: Tweet creation → Event creation → Redis pub/sub → WebSocket delivery → Real-time UI update. Each step happens in milliseconds.

Architecture Overview

Component Architecture

Event Producer Layer: Tweet creation, user actions, system events generate messages
Message Broker Layer: Redis channels route events to interested consumers
Event Consumer Layer: WebSocket handlers deliver events to connected users
Event Store Layer: PostgreSQL stores permanent event records for audit/replay

Control Flow

1. Event Generation: User posts tweet → TweetCreatedEvent generated

2. Event Publishing: Event sent to Redis channel "usertimelineupdates"

3. Event Distribution: All subscribers to that channel receive event

4. Real-Time Delivery: WebSocket connections push event to user browsers

5. Event Persistence: Background process stores event permanently

Data Flow Architecture

Events flow through multiple pathways simultaneously:

• Hot Path: Redis → WebSocket → Browser (millisecond delivery)

• Warm Path: Event Store → Timeline regeneration (second-level consistency)

• Cold Path: Event Store → Analytics processing (minute-level insights)

State Management

Our system maintains multiple state views:

• Connection State: Which users are online, their WebSocket connection IDs

• Subscription State: Which channels each user is listening to

• Event State: Temporary in-memory events awaiting delivery

• Audit State: Permanent event history in database

Implementation Guide

Phase 1: Foundation Setup (15 minutes)

Project Structure Creation

Create the dual-application architecture with proper separation:

# Create project and navigate
mkdir twitter-realtime-lesson4 && cd twitter-realtime-lesson4

# Frontend structure
mkdir -p frontend/src/{components,services,types,hooks,store}
mkdir -p frontend/src/components/{Timeline,Notifications,WebSocket}

# Backend structure
mkdir -p backend/src/{routes,services,models,middleware,types}
mkdir -p backend/src/services/{redis,websocket,events}

Key Dependencies:

• Backend: ws (WebSocket server), redis (pub/sub messaging), uuid (event IDs)

• Frontend: react (UI framework), axios (HTTP client)

Verification: Check both frontend and backend package installations complete successfully.

Phase 2: Backend Real-Time Infrastructure (30 minutes)

Redis Service Implementation

Redis pub/sub operates differently from traditional message queues. Publishers broadcast to channels, subscribers receive all messages on subscribed channels.

// Core pattern for Redis service
class RedisService {
async publishEvent(channel: string, event: Event) {
// Broadcast event to all subscribers of channel
}

async subscribe(channel: string, callback: Function) {
// Register callback for all messages on channel
}
}

WebSocket Connection Management

Connection lifecycle includes authentication, subscription setup, heartbeat monitoring, and graceful cleanup.

// WebSocket service pattern
class WebSocketService {
handleConnection(ws, userId) {
// Authenticate and store connection
// Subscribe to user's channels
// Setup heartbeat interval
}
}

Build Commands:

# Start Redis
docker run -d -p 6379:6379 redis:alpine

# Build and start backend
cd backend && npm run build && npm run dev

Expected Behavior: WebSocket server accepts connections, handles authentication, maintains connection state.

Event Store Implementation

Event sourcing stores every change as an immutable event. Current state is computed by replaying events.

interface Event {
id: string; // Unique identifier
type: EventType; // TWEET_CREATED, USER_FOLLOWED, etc.
payload: any; // Event-specific data
userId: string; // Event originator
timestamp: Date; // When event occurred
version: number; // Event schema version
}

Benefits: Complete audit trail, time-travel debugging, event replay for analytics, natural pub/sub integration.

Phase 3: Real-Time Event Flow (25 minutes)

Tweet Creation and Distribution

Event flow architecture:

1. User posts tweet → TweetCreatedEvent generated

2. Event published to Redis channel tweet_events

3. Event processor identifies followers

4. Individual timeline events sent to each follower's channel

5. WebSocket delivers events to connected users

async function handleTweetCreated(event: Event) {
const followers = getFollowers(event.payload.tweet.authorId);

for (const followerId of followers) {
await publishToChannel(`timeline:${followerId}`, event);
}
}

Testing Commands:

# Test tweet creation API
curl -X POST http://localhost:8000/api/tweets
-H "Content-Type: application/json"
-d '{"content": "Test tweet", "authorId": "user1"}'

Real-Time Notifications

Each user has dedicated channels:

• timeline:${userId} - New tweets for timeline

• notifications:${userId} - User-specific notifications

Performance Consideration: For high-follower users (celebrities), use different distribution strategies to prevent system overload.

Phase 4: Frontend Real-Time Integration (20 minutes)

WebSocket React Hook

Custom hook encapsulates connection management, automatic reconnection, and message handling.

const { isConnected, send } = useWebSocket({
url: 'ws://localhost:8001',
userId: 'user1',
onEvent: handleTimelineUpdate,
onNotification: handleNotification
});

Key Features:

• Automatic authentication on connect

• Heartbeat management with ping/pong

• Exponential backoff reconnection

• Message type routing

Real-Time UI Components

• Timeline Component: Displays tweets with real-time updates

• Notification Panel: Shows live notifications with badges

• Stats Dashboard: Real-time system metrics updated every 5 seconds

Build Frontend:

cd frontend && npm run dev
# Access at http://localhost:3000

Expected Behavior: Real-time tweet updates across browser tabs, live notification delivery, connection status indicators.

Phase 5: Integration Testing and Validation (20 minutes)

Multi-User Concurrent Testing

Test scenarios:

1. Single User: Post tweet, verify it appears in timeline

2. Multi-User: Post from user A, verify followers B and C see it

3. Notifications: User B likes user A's tweet, verify user A gets notification

4. Connection Recovery: Disconnect and reconnect, verify missed events

Load Testing Commands:

# Start complete system
./start.sh

# Run automated demo
./demo.sh

# Manual testing - open multiple browser tabs to http://localhost:3000

Performance Validation

Key metrics to verify:

• Tweet delivery latency < 100ms

• WebSocket connections stable under load

• Event store maintains complete audit trail

• Redis pub/sub handles concurrent distribution

Monitoring Commands:

# Check system stats
curl http://localhost:8000/api/stats

# Monitor Redis activity
redis-cli monitor

Expected Performance:

• 1,000 concurrent WebSocket connections

• Sub-100ms tweet delivery

• Zero message loss during normal operation

• Graceful degradation under extreme load

Phase 6: Production Considerations (10 minutes)

Scaling Patterns

• Horizontal Scaling: Multiple WebSocket server instances with load balancer sticky sessions

• Redis Clustering: For 10,000+ connections, implement Redis Cluster with consistent hashing

• Event Store Optimization: Use PostgreSQL with proper indexing for production

Monitoring and Observability

Key metrics: Connection count, message throughput, event processing latency, error rates.

Real-World Production Considerations

Scaling Patterns

Connection Pooling: Redis supports 10,000+ concurrent connections per instance. For larger scale, we'd use Redis Cluster with consistent hashing to distribute connections.

WebSocket Scaling: Each server instance handles ~1,000 WebSocket connections. Beyond that, we'd need load balancer sticky sessions or connection state sharing.

Event Ordering: Redis pub/sub provides FIFO ordering per channel. For global ordering across channels, we'd need vector clocks or central timestamp service.

Failure Handling

Connection Recovery: Browser reconnects automatically, requests missed events from last known timestamp
Redis Failover: Redis Sentinel provides automatic failover with minimal message loss
Event Replay: Event store enables replaying missed events during outages

Success Criteria

By lesson completion, your system will:

• Deliver new tweets to all online followers within 100ms

• Maintain WebSocket connections for 1,000 concurrent users

• Store complete audit trail of all user actions

• Handle browser refreshes and network interruptions gracefully

• Provide real-time user count and activity metrics

Final Verification Checklist

• Real-Time Tweet Delivery: New tweets appear instantly in followers' timelines

• WebSocket Stability: Connections remain stable with automatic reconnection

• Event Sourcing: Complete audit trail of all user actions

• Notification System: Live notifications for likes and follows

• Performance Metrics: System handles target load with acceptable latency

• Multi-User Testing: Concurrent users see consistent real-time updates

Upon completion, your system demonstrates:

1. Production-Ready Patterns: WebSocket management, event sourcing, pub/sub messaging

2. Real-World Performance: 1,000 concurrent users with < 100ms latency

3. Operational Excellence: Monitoring, logging, graceful failure handling

4. Scalability Foundation: Architecture supports horizontal scaling

Assignment Challenge

Extend the system to support real-time typing indicators (like "X is typing..."). This requires:

• Detecting typing events in UI

• Throttling events to prevent spam

• Broadcasting to conversation participants only

• Implementing timeout for stale indicators

Bonus Challenge: Add real-time collaboration features - multiple users editing the same tweet draft simultaneously (think Google Docs for tweets).

What's Next

Lesson 5 introduces caching strategies that will make our real-time system blazingly fast. We'll implement multi-layer caches that reduce database load by 10x while maintaining real-time responsiveness.

The foundation we're building today - event sourcing and real-time delivery - becomes the backbone for advanced features like trending topic detection, recommendation engines, and abuse prevention systems.

The real-time foundation you've built becomes the backbone for advanced caching strategies in Lesson 5. You'll implement multi-layer caches that reduce database load by 10x while maintaining real-time responsiveness.

Key Integration Points:

• Cache invalidation triggered by real-time events

• Cache warming based on user activity patterns

• Distributed cache consistency across regions

Your event-driven architecture provides the perfect foundation for intelligent caching decisions based on real user behavior patterns.

This lesson transforms your Twitter clone from a static website into a living social platform. Master these patterns, and you'll understand the core architecture behind every major real-time application - from Discord to TikTok to collaborative tools like Figma.