Scalable Notification System — Architecture & Engineering Design

Scalable Notification Platform

Proposed reference architecture • Private engineering design • Version 1.0

Open architecture canvas

1. Requirements and design targets

Support multi-tenant email, SMS, mobile push and in-app notifications; OTP/security, transactional updates and opt-in campaigns; scheduling, localization, quiet hours, preferences, delivery tracking and cancellation.

Assumptions, not benchmarks: 10M logical notifications/day, 1.5 deliveries per notification, 15M channel deliveries/day. Average: 116 requests/s and 174 deliveries/s. A 20× burst requires approximately 3,500 deliveries/s before retries or extra device fanout. Actual traffic, budget, cloud, regions and providers remain open decisions.

Proposed SLOs: 99.95% monthly API availability for valid authorized traffic within agreed tenant limits; overload rejection within those limits counts as failure. p95 durable acceptance under 200ms. Critical queue age p99 under 5s when providers and quotas are healthy. Scheduled jobs become eligible within 60s of their due time. These targets require load validation. Provider acceptance latency and actual carrier/device delivery latency are separate metrics.

Critical, transactional and bulk traffic have separate queues and reserved worker capacity. Bulk traffic must not exhaust critical capacity.

2. Architecture and request flow

Initial AWS reference stack: stateless services on ECS, managed PostgreSQL, SQS standard queues, Redis, S3 archives, interchangeable email/SMS adapters, FCM/APNs and SSE/WebSocket gateways. Use managed services and a single active region spanning availability zones. Kafka becomes an option when long replay retention or many independent consumers justify its operational cost.

1. Producers call authenticated ingress through WAF/API gateway. Validate payload, tenant scope, template, recipient, expiry and limits.
2. In one PostgreSQL transaction, insert notification request and outbox event. Return 202 only after commit.
3. Outbox relay publishes committed events to the ingress queue, marking published afterward. A crash can duplicate publication.
4. Orchestrator applies preferences, quiet hours, template version and channel selection. Persist deterministic delivery rows plus dispatch outbox in one transaction.
5. Dispatch relay routes to queues per channel and priority. Messages contain IDs and small metadata; large payloads stay in DB/object storage.
6. Workers obtain durable leases, recheck consent/suppression/expiry, apply provider and tenant rate caps, persist attempt intent, then call provider adapters.
7. Persist outcomes before acknowledging queue messages. Capability-specific callbacks enter a verified durable webhook inbox and update delivery status through a reconciler. Scheduled requests stay in durable storage until due. Scheduler claims due rows using bounded leases/skip-locked transactions and creates outbox events atomically; it does not rely on memory timers or arbitrary long queue delays. Quiet-hour deferrals must not exceed expiry.

For campaigns, store an audience snapshot/reference, then fan out in bounded pages with checkpoints, tenant budgets and deterministic recipient IDs. A retry of a page must not recreate deliveries.

Database boxes on the canvas are logical table groups, not a requirement for separate clusters. Cross-table outbox transactions must stay within the same database transaction boundary. All outbound channels use the shared reliability controls even where arrows are omitted for readability. Email/SMS callbacks pass through the verified webhook inbox. Push feedback depends on provider capabilities; no universal delivered callback is assumed.

The transactional outbox avoids a database/queue dual-write gap; consumers still require idempotency. AWS outbox guidance

3. Delivery correctness

Internal processing is at least once. External exactly-once delivery is not guaranteed.

Enforce unique (tenant_id, idempotency_key) in PostgreSQL and compare a canonical request hash. Same key and payload returns the original ID; different payload returns 409. Retain dedupe records for at least the published retry/replay window. Redis is only a cache.

Create one delivery per notification, recipient, channel and endpoint where relevant. Include endpoint_id for multiple push devices; use a stable non-null logical endpoint for other channels. A durable unique constraint prevents concurrent duplicate fanout.

Delivery states: SCHEDULED → READY → PROCESSING → PROVIDER_ACCEPTED → DELIVERED when supported. Additional states: RETRY_WAIT, UNKNOWN, FAILED, SUPPRESSED, EXPIRED and CANCELLED. Notification-level status aggregates its deliveries. READ is a separate inbox attribute.

Acquire a lease only for eligible READY jobs. Persist attempt intent before external I/O; update outcomes with attempt ID/version fencing. Keep lease/queue visibility longer than the provider timeout and renew during work. Recovery of expired PROCESSING attempts must check whether provider acceptance is uncertain, then reconcile UNKNOWN before resending. A stale worker must not overwrite a newer result. Fencing cannot retract a provider request already in flight.

Use a stable provider idempotency key where supported. Timeout after sending may mean the provider accepted the message: query status or await evidence before a bounded product-specific retry decision. Do not blindly switch providers on ambiguous timeouts.

Callbacks are deduplicated by provider/event ID, can arrive before synchronous responses, and must never regress DELIVERED to PROVIDER_ACCEPTED. Preserve raw normalized events and use provider-specific transition rules. Provider acceptance alone never proves receipt by a person/device.

4. Retries, scheduling and overload

Use circuit breakers by provider/channel. Route to an approved alternative on definite failures or reconciled outcomes, subject to cost and policy. Re-check consent immediately before dispatch, while documenting that a concurrent opt-out cannot retract a send already accepted by the provider.

Redis token buckets enforce tenant, provider and channel limits. During Redis failure, use conservative bounded local capacity or pause bulk; never silently dispatch without limits. Autoscale on queue age and throughput within provider caps. Sustained quota overload requires backpressure, bulk throttling and admission control; queues alone cannot fix it.

FCM recommends avoiding traffic spikes and respecting retry/backoff signals. Do not assume a fixed quota without checking the actual provider account. FCM scaling guidance

5. Data model

All tenant-owned records and queries enforce tenant scope.

At 1KB per delivery, 15M/day means 15GB/day raw or 450GB/30 days before indexes, replicas and attempts. Suggested starting retention: 7-day hot attempt details, 30-day delivery summaries, configurable encrypted S3 archives. Retention must match business requirements. Prune outbox rows only after safe publication and audit retention; keep dedupe keys long enough.

Partition hot data and benchmark maintenance/write amplification. If write latency/storage headroom fails targets, shard by tenant with outbox transactions local to each shard; alternatively redesign the delivery store around DynamoDB with its own transactional/stream publishing boundary. Do not introduce a new DB/queue dual-write gap during migration.

6. API contract

Tenant identity comes from credentials, never a trusted body field.

POST /v1/notifications Headers: Authorization, Idempotency-Key

{
  "recipient_id": "user_123",
  "template_id": "order_shipped",
  "template_version": 3,
  "channels": ["email", "push"],
  "category": "transactional",
  "priority": "transactional",
  "variables": {"order_id": "ord_456"},
  "schedule_at": null,
  "expires_at": "2026-10-01T12:00:00Z"
}

202 response: notification_id, status="ACCEPTED", status_url. Expiry shown is illustrative; clients must supply a valid future timestamp.

7. In-app and mobile delivery

Inbox worker commits inbox item and realtime outbox together. Relay notifies SSE/WebSocket gateways; pub/sub is an optimization. Clients reconnect and fetch the durable inbox using a cursor. Track read state per user; authorize every subscription. No message is lost merely because a socket disconnects.

Push endpoint fanout is deterministic per device. Remove invalid device tokens and account for multiple devices in capacity estimates. Device OS policies and user permissions influence notification visibility independently of backend acceptance.

8. Capacity and scaling

Average deliveries: 15,000,000 / 86,400 ≈174/s; 20× burst ≈3,500/s excluding retries. At a hypothetical 200ms average provider request time, concurrency = 3,500 ×0.2 =700 in-flight calls; provision roughly 1,400 for 2× headroom only if quotas and load tests support it. Size each channel separately.

Backlog drain time = backlog / (service rate − incoming rate). A 1M backlog with 5,000/s service and 3,500/s incoming drains in about 667s, assuming quotas permit that service rate. If service rate is not greater than arrival rate, backlog never drains.

Keep autoscaling bounded by provider quota and spend caps. Reserve critical worker pools, fairly schedule tenants, smooth campaigns and expire obsolete OTPs. Measure database IOPS, connection pools, queue lag, retry amplification and gateway connection counts before scaling.

9. Deployment, security and disaster recovery

Run API, relay, scheduler and workers across availability zones; use managed database failover and durable queues. Start with one active sending region. Warm standby needs an ownership epoch/fencing mechanism and a verified old-region shutdown before promotion to avoid split-brain sends.

Proposed DR targets: RPO ≤5min and RTO ≤30min, conditional on replication and restore drills. Asynchronous regional replication can lose acknowledged notifications during a regional disaster; document that tradeoff and reconcile uncertain provider attempts after failover.

Use TLS, encryption at rest, secret manager and rotated provider credentials; restrict network and IAM permissions. Encrypt addresses/tokens, redact PII/OTP/payloads from logs, escape templates, validate webhook signatures/timestamps and prevent replay. Apply tenant RBAC, audit admin/replay actions and spend caps. Opt-out/transactional exceptions require explicit category policy; no blanket consent bypass.

10. Observability

Track API availability/acceptance latency; oldest unpublished outbox age; queue age per channel/priority; scheduler lateness; worker saturation; provider 429/error/success; UNKNOWN age; retry/DLQ rates; suppressions/bounces; delivery evidence latency; tenant fairness and spend.

Trace notification_id → delivery_id → attempt_id. Avoid user IDs as metric labels. Alert on critical queue age, stuck outbox, prolonged UNKNOWN, quota saturation, DLQ growth, high bounce rates and database capacity. Dashboards distinguish provider acceptance from confirmed delivery.

11. Failure recovery matrix

12. Validation plan

Test concurrent identical idempotency requests and mismatched payloads; deterministic paged fanout; tenant isolation; consent/cancel/expiry races; duplicate and out-of-order callbacks; stale leases; ambiguous provider acceptance; safe DLQ replay.

Inject crashes at DB commit, outbox publish, provider call and ACK boundaries. Run 20× burst and soak tests with provider stubs, then validate actual provider quotas with controlled integration tests. Test backlog recovery, Redis failure, database restore and regional fencing. Verify neither generated diagrams nor capacity formulas are treated as production test results.

13. Implementation roadmap and open decisions

1. Build API, PostgreSQL schema/outbox, SQS ingress, email worker, durable idempotency, preferences and inbox.
2. Add SMS/push, scheduler, retry/DLQ, provider feedback, cancellation and UNKNOWN reconciliation.
3. Add campaigns, tenant fairness, reserved critical capacity, dashboards, rate caps and load-driven autoscaling.
4. Validate partitioning/sharding and warm-standby DR through measured capacity and recovery drills. Confirm actual volume and burst shape, channel mix/device fanout, providers, cloud, budget, regions, retention, tenant isolation requirements and critical delivery SLOs before production sizing.