Lesson 20 :External Integration

Hands On System Design with Distributed Systems Implementation ~8 min read 60 min hands-on

BridgeGate External Hub: Multi-Channel Alert Fan-Out

Your log platform detects a payment outage at 2 AM. Who needs to know? Slack for the team chat, email for the exec summary, PagerDuty for the on-call engineer, JIRA for the ticket, a webhook for your custom runbook, S3 for the audit archive, and Prometheus so Grafana shows the spike. BridgeGate External Hub composes all seven outbound channels into one FastAPI orchestrator on port 8200: ingest an event once, evaluate severity-based routing rules, fan out to every selected engine in parallel, persist delivery records in SQLite, and stream live KPI updates to a React dashboard over WebSocket.

Core Components

Slack Notification Engine

Role: Deliver team chat alerts with deduplication and per-minute rate limiting.

SlackEngine in services/slack/src/engine.py fingerprints events by service:severity:message (SHA-256, truncated), enforces a configurable dedup window (default 60 seconds) and rate limit (default 30/minute), and posts a structured payload to the configured webhook URL. When BRIDGEGATE_INTERNAL_DELIVERY=1 (default), deliveries land in the embedded /internal/slack-inbox endpoint so counters reflect real HTTP posts, not simulated increments.

Email Alert Engine

Role: Send triggered HTML alerts and scheduled summary reports via SMTP.

EmailEngine in services/email/src/engine.py renders Jinja2 templates for per-event alerts and 24-hour platform reports, then delivers through SMTP. Docker Compose routes to MailHog (mailhog:1025); local runs default to localhost:1025 with a file fallback that writes .eml files to data/mailbox/ when SMTP is unreachable.

On-Call Incident Engine

Role: Route CRITICAL and ERROR events to PagerDuty or OpsGenie by escalation policy.

IncidentEngine in services/incident/src/engine.py maps severity to provider — CRITICAL → PagerDuty, ERROR → OpsGenie by default — and emits Events API v2–shaped payloads with routing_key, dedup_key, and incident_id. Incidents are tracked in memory and exposed via /api/incidents.

Ticketing Engine

Role: Auto-create JIRA or ServiceNow tickets on high-severity events with dedup by severity threshold.

TicketingEngine in services/ticketing/src/engine.py tickets only when severity is in auto_ticket_severities (default ERROR and CRITICAL). CRITICAL events open JIRA keys (LOG-XXXXXXXX); ERROR events open ServiceNow keys (INCXXXXXXXX). Rows persist in the tickets SQLite table and surface on the dashboard and /api/tickets.

Webhook Delivery Engine

Role: Filter-match events to registered endpoints, render Jinja2 JSON templates, and retry failed deliveries.

WebhookEngine in services/webhook/src/engine.py stores endpoints and delivery history in SQLite, matches events against field filters (equals, contains, regex), renders template strings to JSON via _render(), and retries up to max_retries with linear backoff. A default endpoint is seeded on first boot.

S3 Object Export Engine

Role: Partition and export event JSON to MinIO or local object storage.

S3ExportEngine in services/s3export/src/engine.py buffers events, builds keys from date/service partitions under the configured prefix (logs/), and uploads via boto3 when S3_ENDPOINT is set or writes to data/exports/s3/ locally. Batch export replays the 50 most recent routed events from SQLite.

Metrics Export Engine

Role: Expose Prometheus counters and push aggregated stats to a Datadog intake endpoint.

Keep going with the full lesson

The rest of this lesson — plus its implementation guide, runnable scripts and source code — is included with the course or an All Access plan. Create a free account to start the free lessons in every course.

Already a member? Sign in