Day 2: Tweet Storage and Retrieval

Building the Heart of Social Media Communication

What We're Building Today

Today we're constructing the core engine that powers every social media platform. You'll build a production-ready system that handles the fundamental operations every user takes for granted: posting tweets, editing them, and seeing real-time engagement updates.

Key Deliverables:

• Tweet posting API with media attachment support

• Tweet versioning system for edit functionality

• Real-time engagement tracking (likes, retweets, replies)

• Performance-optimized REST endpoints handling 100 tweets/second

• Professional monitoring dashboard showing system metrics

Core Concepts: The Tweet Lifecycle Engine

Tweet Immutability vs. Editability Paradox

Traditional databases assume data changes through updates. Social media breaks this assumption - tweets need to be both permanent (for legal and audit reasons) and editable (for user experience). We solve this through event sourcing, where each tweet edit creates a new version while preserving the complete history.

Think of it like Google Docs revision history, but optimized for millions of concurrent users.

Content-Addressable Storage Pattern

Each tweet gets a unique content hash, enabling instant duplicate detection and efficient storage. When someone posts identical content, we reference the existing stored version rather than creating duplicates - a technique Twitter uses to save petabytes of storage.

Engagement Velocity Tracking

Raw engagement counts lie. A tweet with 1000 likes in 5 minutes indicates viral potential, while 1000 likes over 5 days suggests normal performance. We track engagement velocity (rate of change) to power recommendation algorithms and trending detection.

Context in Distributed Systems

Position in Twitter Architecture

Our tweet storage sits between the user interface and timeline generation services. It must handle burst traffic during breaking news while maintaining consistency for the timeline algorithms we'll build next week. The API design directly impacts how quickly users see new content and how efficiently we can generate personalized feeds.

Real-time Production Requirements

Instagram processes 400 million stories daily, requiring sub-millisecond tweet retrieval. TikTok's recommendation engine requests tweet metadata 50,000 times per second. Your storage layer must satisfy these demands while supporting complex queries for search, analytics, and content moderation systems.

Architecture: Three-Tier Storage Strategy

[Insert Component Architecture Diagram Here]

Tier 1: Hot Storage (Redis)

Active tweets from the last 24 hours live in Redis for instant access. This powers real-time timelines and engagement tracking. Each tweet stores core metadata plus engagement counters that update atomically.

Tier 2: Warm Storage (PostgreSQL)

Complete tweet data with full-text search capabilities. Houses tweet content, media references, version history, and relationship data. Optimized indexes enable complex queries for user profiles and hashtag tracking.

Tier 3: Cold Storage (Object Storage)

Media files and archived tweet versions move to S3-compatible storage. Content-addressed filenames prevent duplication while CDN integration ensures global delivery performance.

Data Flow: Write Path

[Insert Data Flow Diagram Here]

1. Client posts tweet through REST API

2. Content validation and media processing

3. Atomic write to PostgreSQL with version creation

4. Cache population in Redis hot storage

5. Asynchronous media upload to object storage

6. Event emission for timeline generation

Data Flow: Read Path

1. API request with tweet ID or query parameters

2. Cache check in Redis hot storage

3. Database fallback with optimized queries

4. Media URL resolution from object storage

5. Response assembly with engagement data

State Management: Tweet Lifecycle

[Insert State Machine Diagram Here]

Tweets transition through states: Draft → Published → Edited → Archived. Each state change creates immutable log entries while maintaining a current state pointer. This enables instant rollbacks and complete audit trails.

The engagement state operates independently - likes, retweets, and replies update through separate atomic operations that don't affect the core tweet content state.

Implementation: Building Your Tweet Storage System

Phase 1: Project Foundation (15 minutes)

Initialize Modern React Project

# Create project with latest Vite + TypeScript
npm create vite@latest twitter-tweet-storage -- --template react-ts
cd twitter-tweet-storage

Install Production Dependencies

# Core libraries - React 18.3+, TypeScript 5.4+
npm install @tanstack/react-query axios lucide-react date-fns nanoid zod

# Backend - Express 4.19+, latest middleware
npm install express cors multer uuid redis pg

# Development tools - Vitest, Testing Library
npm install -D @testing-library/react @testing-library/jest-dom vitest jsdom

The latest Vite template provides optimal build performance and TypeScript integration. We choose @tanstack/react-query over older alternatives for superior caching and synchronization.

Phase 2: Type System Architecture (20 minutes)

Design Core Data Structures

Create src/types/tweet.ts with our fundamental data structures:

// Key insight: Separate engagement from content for performance
interface Tweet {
  id: string;            // Unique identifier using nanoid
  content: string;       // Max 280 characters
  version: number;       // Incremental version tracking
  engagement: TweetEngagement; // Separate object for atomic updates
}

interface TweetEngagement {
  likes: number;
  retweets: number;
  replies: number;
  views: number;
  likedByCurrentUser: boolean;
  retweetedByCurrentUser: boolean;
}

Why This Matters: Separating engagement from core tweet data enables atomic counter updates without affecting main content, crucial for high-concurrency scenarios.

Phase 3: Backend API Architecture (45 minutes)

Express Server with Performance Monitoring

Create src/api/server.ts with comprehensive middleware:

• CORS configuration for localhost development

• Request/response timing middleware

• Graceful shutdown handling

• Health check endpoint returning uptime metrics

Tweet Storage Model Implementation

In src/api/models/Tweet.ts, implement the core storage patterns:

// Use Map for O(1) lookups, critical for performance
private static tweets: Map<string, Tweet> = new Map();
private static engagement: Map<string, TweetEngagement> = new Map();

Why Maps Over Arrays: Map provides O(1) retrieval by ID versus O(n) array searching. Critical for timeline generation performance.

Atomic Engagement Updates

// Thread-safe engagement updates
updateEngagement(tweetId, action) {
  const engagement = this.engagement.get(tweetId);
  // Atomic increment prevents race conditions
  engagement.likes += (action === 'like' ? 1 : -1);
}

Production Insight: Real systems use Redis INCR/DECR for atomic counter operations. Our in-memory approach demonstrates the pattern.

API Routes Implementation

Create comprehensive REST endpoints in src/api/routes/tweets.ts:

• POST /api/tweets - Create tweet with media support

• GET /api/tweets - List tweets with filters and pagination

• PUT /api/tweets/:id - Update tweet content

• POST /api/tweets/:id/engagement - Atomic engagement updates

• GET /api/tweets/:id/versions - Version history retrieval

Phase 4: Frontend Component Architecture (35 minutes)

Tweet Form with File Upload

Build src/components/TweetForm/TweetForm.tsx with these key patterns:

• FormData for multipart uploads

• Character count with visual feedback

• Optimistic UI updates

• Error boundary integration

Real-Time Engagement System

In src/components/Tweet/TweetCard.tsx, implement:

• React Query for automatic cache invalidation

• Optimistic updates for immediate feedback

• Mutex pattern preventing double-clicks

• Visual state transitions for user feedback

Performance Dashboard Components

Create monitoring components in src/components/Dashboard/:

• useEffect hooks for periodic metric collection

• Chart components using CSS transforms

• Real-time RPS calculation

• Memory usage display with formatting

Phase 5: Testing and Validation (25 minutes)

Unit Testing Setup

# Run component tests
npm test
# Expected: All tests pass, coverage > 80%

Critical Test Cases:

• Tweet creation with validation

• Engagement atomic updates

• Character limit enforcement

• Error handling scenarios

Performance Validation

# Test API endpoints
curl -X POST http://localhost:3001/api/tweets 
  -H "Content-Type: application/json" 
  -d '{"content":"Test tweet","authorId":"user1","authorUsername":"testuser"}'
# Expected: 200 response with tweet object in < 50ms

Load Testing Simulation

# Simulate high load - 100 concurrent tweets
for i in {1..100}; do
  curl -X POST http://localhost:3001/api/tweets 
    -H "Content-Type: application/json" 
    -d '{"content":"Load test '$i'","authorId":"load","authorUsername":"loadtest"}' &
done
wait
# Expected: All requests complete successfully under 100ms

Phase 6: Production Readiness (15 minutes)

Docker Configuration

Create docker-compose.yml with:

• Multi-stage builds for optimized images

• Health check endpoints

• Graceful shutdown handling

• Volume mounting for development

Performance Monitoring

# Access monitoring dashboard
curl http://localhost:3001/api/tweets/system/stats
# Expected: Detailed system metrics including memory, uptime, performance

Performance Optimization Secrets

Write Amplification Mitigation

Each tweet write triggers multiple storage operations (cache, database, object store). We use write-behind caching and batch operations to minimize I/O overhead while maintaining consistency guarantees.

Read Pattern Optimization

90% of tweet reads happen within 24 hours of posting. Our tiered storage places fresh content in fastest storage while older content gracefully degrades to slower but cheaper storage.

Engagement Counter Architecture

Instead of updating database rows for each like, we use Redis counters with periodic batch writes to persistent storage. This reduces database load by 95% while providing real-time engagement feedback.

Production Insights

Media Processing Pipeline

Instagram generates 15 different image sizes for each upload to optimize mobile performance. We implement similar multi-resolution processing with progressive JPEG encoding and WebP conversion for modern browsers.

Version History Compression

Twitter stores edit histories efficiently through delta compression - only changes between versions are stored, not complete duplicates. This reduces storage costs by 80% for heavily edited tweets.

Global Consistency Challenges

When a tweet goes viral globally, engagement updates arrive from multiple continents simultaneously. We use eventually consistent counters with conflict-free replicated data types (CRDTs) to handle concurrent updates without data loss.

Success Validation Checklist

Functional Requirements:

• Tweet creation with 280 character limit

• Media attachment support (images/videos)

• Edit functionality with version tracking

• Real-time engagement updates

• Timeline retrieval with pagination

Performance Requirements:

• Response time < 100ms (P95)

• Throughput > 100 tweets/second

• Concurrent user support

• Memory efficient engagement tracking

Production Requirements:

• Error handling and validation

• Monitoring and observability

• Graceful degradation

• Docker containerization

• Test coverage > 80%

Common Implementation Pitfalls

Race Conditions in Engagement

// Wrong: Non-atomic updates
engagement.likes = engagement.likes + 1;

// Correct: Atomic operations
engagement.likes += 1; // Single operation

Memory Leaks in React Components

// Wrong: Missing cleanup
useEffect(() => {
  const interval = setInterval(fetchData, 1000);
}, []);

// Correct: Cleanup on unmount
useEffect(() => {
  const interval = setInterval(fetchData, 1000);
  return () => clearInterval(interval);
}, []);

Inefficient Database Queries

// Wrong: N+1 query pattern
tweets.forEach(tweet => getEngagement(tweet.id));

// Correct: Batch operations
const engagements = getEngagementsForTweets(tweetIds);

Testing and Validation

Your implementation must pass these production criteria:

• Handle 100 tweets/second sustained load

• Serve read requests under 100ms (P95)

• Support concurrent engagement updates without race conditions

• Maintain data consistency during peak traffic bursts

• Gracefully handle media upload failures

Deployment Verification

# Run full system test
./scripts/demo.sh
# Expected: All demo scenarios complete successfully

Integration Points

This lesson builds on last week's database schema and prepares for next week's timeline generation. The tweet IDs you generate here become the primary keys for timeline ordering, while engagement metrics directly feed the recommendation algorithms we'll implement in Week 3.

The storage patterns learned here scale to billions of tweets with proper database selection and caching strategies covered in upcoming lessons.

Success Metrics

By completing this lesson, you'll have built the core storage engine that powers platforms serving millions of users. Your API will handle real-world traffic patterns while maintaining the data consistency required for financial and legal compliance in production social media systems.

Expected Time Investment:

• Phase 1-2: 35 minutes (Foundation)

• Phase 3: 45 minutes (Backend)

• Phase 4: 35 minutes (Frontend)

• Phase 5: 25 minutes (Testing)

• Phase 6: 15 minutes (Production)

• Total: 2.5 hours for complete implementation

Upon completion, you'll have mastered production-ready patterns for social media storage that directly apply to any high-scale content management system. The architecture patterns learned here form the foundation for everything we'll build in the remaining weeks of this course.