Lesson 10: Security and Compliance — Protect logs before they become liability

Hands On System Design with Distributed Systems Implementation ~6 min read 60 min hands-on

What we're building today

When Datadog ingests payment logs, access is not a free-for-all—RBAC gates who sees what. Stripe encrypts PAN fields at ingest; Splunk redacts before analysts search. GDPR erasure in Shopify removes a buyer's trail across shards. This lesson wires seven security layers into one platform.

By the end you'll have:

  • JWT authentication with RBAC on log namespaces

  • AES-256-GCM field encryption for PII

  • Regex redaction before display

  • Hash-chained immutable audit trails

  • Automated retention policies

  • GDPR right-to-erasure workflow

  • SOX / HIPAA / GDPR compliance report export

Why this matters

  • AWS CloudTrail records who touched which resource—your audit chain mirrors that immutability model.

  • HashiCorp Vault encrypts secrets at the field level—same idea as encrypting email and ssn in log JSON.

  • Elastic Security redacts PAN before indexing—your redaction engine runs on the read path.

  • SOC 2 auditors ask for retention proof—your policy engine deletes or archives on schedule.

  • Meta GDPR tooling erases user data across stores—your erasure coordinator tracks locations and verifies completion.

Security is not a checkbox—it is a pipeline every log byte passes through.

Architecture

Single FastAPI service on port 8094. All security modules run in-process with SQLite persistence.

How data moves through the system

Ingest path: Authenticated user posts log JSON → RBAC checks logs:write:<resource> → sensitive fields encrypted → stored in SQLite → audit WRITE.

Read path: RBAC checks logs:read:<resource> → load records → admin may see decrypted; analyst gets redacted view → audit READ with hash chain link.

Compliance path: Retention applies policies → GDPR erasure removes by user_id → report endpoint aggregates audit stats + retention + erasure into exportable JSON.

Project layout

Clone or open week10-security-compliance-platform/:

Design highlights

  • SQLite + lock: API and background log generator share one database safely.

  • Encryption keys in memory: After a restart, older ciphertext may not decrypt—production systems use a key vault; here you see why rotation matters.

  • RBAC wildcards: Policies like logs:read:application.* teach IAM-style matching without a cloud account.

  • Dashboard via CDN: React loads from CDN—no node_modules or npm build step.

  • Secrets: Set JWT_SECRET_KEY in your shell before starting. Never commit real signing keys. Demo passwords are for local practice only.

Keep going with the full lesson

The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.

Already a member? Sign in