Sweaty Imposter

Design a Chat App (WhatsApp/Messenger)

System DesignReal TimeWrite HeavyMid3 min readUpdated

Messages have to arrive in order, survive a dropped connection, and reach phones that are offline. The design hinges on how a message finds the gateway node holding the recipient's socket.

1 · Requirements

The Prompt

Design 1:1 and group messaging with delivery receipts, online presence, and offline message storage.

Functional

  • Send 1:1 and group messages
  • Show sent, delivered, and read receipts
  • Show who is online
  • Deliver messages that arrived while a device was offline
  • Send photos and other media

Non-Functional

  • Messages arrive in order within a conversation
  • A message is never lost once the sender sees it as sent
  • Delivery to an online recipient takes well under a second

2 · Estimates

Drill This Phase First

Show the Answer and Estimates

Answer

About 50,000 nodes

20% of 500M is 100M open sockets. 100M divided by 2,000 sockets a node is 50,000 nodes.

Writes
500M daily users sending 40 messages a day: 20B messages, about 230,000 a second and 600,000 at peak
Sockets
About 20% of daily users connected at once: 100M open WebSockets across about 50,000 gateway nodes
Storage
About 200 bytes a message: roughly 4 TB a day and 1.5 PB a year in a wide-column store
Media
10% of messages carry about 500 KB: roughly 1 PB a day to object storage, and CDN egress dominates bandwidth
Presence
100M heartbeats every 30 seconds: about 3.3M operations a second into a Redis presence cluster
Groups
Average group size is 20, but most messages are 1:1, so write amplification is only about 1.05x

3 · API

WS /connect (auth header), bidirectional message + receipt frames
POST /messages { conversation_id, body, client_msg_id } → { message_id, ts }
GET /conversations/:id/messages?before=ts → { messages[] }
POST /media (multipart) → { media_id, url } (then attach to message)
POST /receipts { message_id, status: delivered|read } → 204

Data Model

messages (wide-column, partition = conversation_id, cluster = ts):
  conversation_id (PK) : uuid
  ts                   : timeuuid
  sender_id            : uuid
  body                 : string
  media_id             : uuid?
  status               : enum(sent|delivered|read)

conversations (KV):
  conversation_id (PK) : uuid
  type                 : enum(direct|group)
  member_ids           : list<uuid>
  last_msg_ts          : timestamp

presence (Redis, TTL 60s):
  user_id              : string
  node_id              : string
  last_seen            : timestamp

4 · High-Level Design

A sent message goes from the sender's gateway to the chat service and into Kafka. Workers persist it, then look up the recipient's gateway in Redis presence to deliver it live, or send a push notification when the recipient is offline. History reads come from Cassandra by conversation.

5 · Deep Dives

Routing to the Right Socket

Each gateway registers user_id to node_id in Redis presence with a short TTL. To deliver, look up the recipient's node and forward the frame there. When the entry is missing, the user is offline: store the message and send a push notification.

Ordering and Duplicates

Partition Kafka by conversation_id so one consumer sees a conversation's messages in order. Clients attach a client_msg_id, so a retry after a dropped connection gets deduplicated on the server and never shows twice.

Receipts at Scale

Delivered and read receipts are messages too, and in a group of 20 each message can produce 40 of them. Batch receipts per conversation and send only the latest read position, which keeps receipt traffic close to the message rate.

6 · Wrap Up

Name what you would tackle next: end-to-end encryption and what it does to server-side search, multi-device sync for one account, and draining gateway nodes during a deploy without dropping sockets.

The Short Version

Persistent WebSocket per client; message queue for offline delivery; wide-column store keyed by (conversation_id, ts). Object storage for media attachments.

Building Blocks