Lesson 10: Security and Compliance

Secure Log Access and Compliance: Protecting PII from Ingest to Erasure

Summary

Logs quietly accumulate personal data: emails in error messages, card numbers in debug output, tokens in request dumps. This system treats log storage as a regulated datastore. It redacts free text and encrypts PII fields before events leave the ingest edge, enforces role- and team-scoped reads with per-field reveal rights, writes every security decision to a tamper-evident audit chain, applies per-category retention, erases data subjects on request, and exports sealed compliance reports.

Core Concepts

Redaction vs encryption. Redaction is irreversible: SSNs become [REDACTED:SSN], IPs collapse to a /24, cards keep their last four digits. Encryption is reversible for authorised roles. Free text is redacted because you cannot reliably find every field in it; structured PII fields are encrypted because some roles legitimately need them back.

Envelope encryption. A master key wraps versioned AES-256 data keys. Fields are encrypted with AES-GCM using eventId:fieldName as associated data, binding each ciphertext to its location. Rotation creates a new data key; old versions remain for decryption.

Blind index. An HMAC of the normalised value under a separate key allows exact-match search on encrypted fields without decrypting them. Equality is the only thing it leaks.

Hash-chained audit. Each record stores the previous record's hash. Changing or removing any row breaks every hash after it. A database trigger rejects UPDATE, DELETE and TRUNCATE as a second layer.

Forward-only saga. Erasure runs as idempotent steps that can be retried from the start, avoiding compensation logic.

Context in Distributed Systems