Sweaty Imposter

Design a Notification System

System DesignFan-OutReliabilitySenior3 min readUpdated

Push, email, and SMS each have their own providers, quotas, and failure modes. The design keeps one slow provider from blocking the others and makes sure nobody gets the same alert twice.

1 · Requirements

The Prompt

Send push, email, and SMS notifications to millions with per-user preferences, deduplication, and retries.

Functional

  • Send notifications by push, email, and SMS from one API
  • Respect each user's channel preferences and quiet hours
  • Deduplicate repeat requests
  • Retry failures and record delivery status

Non-Functional

  • One failing provider never delays the other channels
  • Each dedup key sends at most one notification
  • Handles about 60,000 sends a second at peak

2 · Estimates

Drill This Phase First

Show the Answer and Estimates

Answer

About 23,000 a second

200M times 10 is 2B notifications a day. 2B divided by 86,400 seconds is about 23,000 a second, and about 60,000 at peak.

Sends
200M users at about 10 notifications a day: 2B a day, about 23,000 a second and 60,000 at peak
Channels
About 70% push, 20% email, and 10% SMS: 16,000 push, 4,600 email, and 2,300 SMS a second
Quotas
APNs limits connections and Twilio limits each number's send rate, so each provider gets its own worker pool and token bucket
Dedup
24 hours of user and key hashes: about 2B entries at 24 bytes, roughly 50 GB in Redis
Retries
About 2% transient failures with up to 5 retries: plan for about 5% extra queue traffic
Prefs
200M users at about 200 bytes each: about 40 GB in a key-value store, sharded by user_id

3 · API

POST /notify { user_id, template_id, vars, channels[], dedup_key? } → { notif_id, status: queued }
GET /notifications/:id → { channel_attempts[], final_status }
PUT /users/:id/preferences { channels: { push: true, email: false }, quiet_hours } → 204
POST /devices/register { user_id, platform, token } → 204
Webhook: POST /providers/:name/callback (delivery receipts) → 200

Data Model

notifications (Cassandra, partition = user_id, cluster = ts DESC):
  notif_id (PK)   : uuid
  user_id         : uuid
  template_id     : string
  channel         : enum(push|email|sms)
  status          : enum(queued|sent|delivered|failed|suppressed)
  attempts        : int
  dedup_key       : string?
  created_at      : timestamp

user_prefs (KV, key = user_id):
  channels        : map<channel, bool>
  quiet_hours     : { start, end, tz }
  topics_muted    : list<string>

dedup (Redis, key = "dd:{user_id}:{dedup_key}", TTL 24h):
  value           : notif_id

4 · High-Level Design

A request is checked against preferences and dedup keys, then queued on its channel's topic. Each channel's workers send through their provider behind a circuit breaker, retry with backoff, and write the result to the delivery log.

5 · Deep Dives

Bulkheads and Circuit Breakers

Give each channel its own topic and worker pool, so a Twilio outage backs up SMS while push keeps flowing. Wrap each provider in a circuit breaker that stops calling after repeated failures and probes again after a cooldown, while messages wait in Kafka.

Deduplication

Producers send a dedup key with each request. Before sending, a worker sets dd:{user_id}:{dedup_key} in Redis only if it is absent, with a 24 hour TTL. A retry from the producer or a redelivered Kafka message then finds the key and skips the send.

Preferences and Quiet Hours

Check preferences at send time, since a user may opt out between queueing and sending. Hold notifications that land in quiet hours in a delayed queue keyed to the user's time zone, and drop them if they expire first.

6 · Wrap Up

Name the next risks: priority lanes so a password reset never waits behind a marketing batch, provider failover for email and SMS, and per-user limits so one chatty service cannot flood anyone's phone.

The Short Version

Pub/sub fan-out by channel; per-channel worker pools (bulkhead); circuit breaker on external providers (APNs/FCM/Twilio); dedup key in a key-value store.

Building Blocks