System design deep dive · HLD
Design WhatsApp: full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for online delivery and offline queuing, and an entity-relationship diagram - with persistent WebSocket connection routing, delivery/read receipts, and end-to-end encryption worked through in detail.
1. Clarify requirements before drawing any box
WhatsApp is a connection-routing problem more than a storage problem: hundreds of millions of clients hold a long-lived connection open, and the system's job is knowing, at any instant, which server a given user is connected to - or safely queuing for them if they aren't connected at all.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
The number that shapes this design is not messages/day - it's concurrent open connections, because every one of them is a small amount of held server memory that has to be routable.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Messages sent (writes) | 100B messages/day across 2B MAU | ~1.15M writes/sec average, ~3M/sec peak (New Year's Eve-class events) |
| Concurrent connections | ~500M users online at any given moment | ~500M held WebSocket connections, spread across the connection-server fleet |
| Connection server capacity | ~500K idle connections per server (each connection is a few KB of memory/state) | ~1,000 connection servers needed for steady-state, more with headroom for failover |
| Offline queue storage | ~5% of recipients offline at send time, message ~1KB incl. metadata | 100B/day × 5% × 1KB ≈ 5 TB/day of transient queued messages |
| Message store (short retention) | Delivered messages deleted from server soon after ack; media stored briefly for retry | Server-side storage stays a rolling buffer, not a permanent archive - the archive lives on-device |
3. High-level design (HLD)
The HLD centers on the connection server as the one component that must be stateful, and a presence registry that lets any connection server find any user's current connection without a broadcast.
What each box owns
Connection servers
Hold one long-lived WebSocket (or custom binary protocol) connection per online client and register their own (user_id → server_id) mapping in the presence registry on connect, removing it on disconnect. They are the only stateful tier in the system and are sharded purely by which clients happen to be connected to them, not by any data-partitioning scheme.
Presence registry
A fast key-value store (Redis-backed) mapping every online user_id to the specific connection server instance holding their socket. Any connection server, on receiving a message for a recipient, looks up this registry to find where to forward it - this is what lets the fleet route messages without an all-to-all broadcast.
Message service and offline queue
Persists every message briefly (for retry/multi-device delivery) and decides the routing outcome: if the recipient is online, forward immediately through their connection server; if offline, append to a per-recipient durable queue and trigger a push notification so the OS wakes the app or shows an alert.
Push notification service
A thin adapter over Apple Push Notification service and Firebase Cloud Messaging. It never carries the actual message content (that stays end-to-end encrypted and is only fetched once the app reconnects) - it only wakes the device so it can open its own connection and pull the queued message.
4. Detailed architecture diagram
The architecture diagram shows how presence and connection routing actually shard, and where group messages fan out to potentially hundreds of recipients without becoming hundreds of separate client round-trips.
| Decision | Choice | Reasoning |
|---|---|---|
| Connection tier state | Sharded stateful servers, sticky per socket | A WebSocket can't be load-balanced per request; each shard just holds whichever clients happened to connect to it. |
| Presence lookup | Centralized fast key-value registry | Any connection server needs to route to any other in O(1) without a broadcast to every shard. |
| Group fan-out | Parallel per-member presence lookups + routing | Groups are capped at a few hundred members, so parallel fan-out is bounded and fast, unlike a social-feed follower list. |
| Encryption | End-to-end (Signal protocol), keys never touch servers | The message service and connection servers only ever handle ciphertext; a server breach cannot expose message content. |
5. Sequence diagrams for the two critical flows
The two flows below are the two branches of the same decision: is the recipient's socket currently open somewhere in the fleet, or not.
5.1 Send a message, recipient online
Step 4's single check mark fires as soon as the message is durably persisted server-side, before it has even reached the recipient - it means "the server has it," not "they got it." Step 7's double check mark only appears once the recipient's device has actually acknowledged receipt, which is why the two states are visibly different in the client UI.
5.2 Send a message, recipient offline (queued + push)
Because the presence lookup in step 2 finds nothing, the connection server never even attempts to forward the message live - it goes straight to durable queuing. The push in step 5/6 carries no message content by design (it's just ciphertext the server can't read anyway); the recipient's app is the one that opens a connection and drains its own queue in step 7, which is also the moment the sender's UI finally shows double-check delivered.
6. Entity-relationship (ER) diagram and schema
The data model has to represent both 1:1 and group conversations uniformly, and track per-recipient delivery/read state separately from the message itself, since a group message has one status per member.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| Wide-column (Cassandra) for messages | Extremely high write volume, access pattern is always by conversation_id + time. | No ad hoc cross-conversation search server-side; that has to be a client-side, on-device index. |
| Relational for message_status | You want simple per-recipient state transitions and easy "unread count" queries. | A large group's status rows multiply per message; needs efficient batched writes, not one row at a time. |
7. Deep dives interviewers actually probe
How does end-to-end encryption actually work if the server routes every message?
Each device holds a long-term identity key pair and generates fresh one-time pre-keys; the Signal protocol's double-ratchet algorithm derives a new symmetric key for every message from a continuously advancing key chain, using key material exchanged out-of-band via the key server's public bundles. The connection servers and message service only ever see and store ciphertext - they route by conversation_id and recipient_id, neither of which requires reading the payload. Group messages are encrypted once per recipient (a "sender key" scheme) so the server still can't decrypt even a message with 200 recipients.
How is a connection server chosen and found again after a client reconnects on a flaky network?
On connect, the client is routed (via consistent hashing or simple least-loaded assignment) to one connection server, which immediately writes its own address into the presence registry keyed by user_id, with a short TTL refreshed by a heartbeat. A dropped connection lets the TTL expire quickly, so other servers stop trying to route to a now-stale address within seconds rather than retrying against a dead socket.
How do group messages avoid becoming N separate client-perceived sends?
The sender's client encrypts and sends the message once to the Message service, which then fans out to each of the (up to a few hundred) member's connection servers in parallel - conceptually identical to the celebrity fan-out problem in social feeds, but bounded and cheap because group size is capped, unlike an unbounded follower list.
What happens on multi-device (phone + linked desktop/web client)?
Each device has its own identity keys and its own entry in the presence registry, so a message is actually fanned out to every one of the sender's own linked devices as well as the recipient's, keeping all clients in sync. This is why message_status and the encryption scheme are modeled per-device under the hood, even though the product surface shows a single "delivered" checkmark per human recipient.
What's the single biggest bottleneck as this scales 10x?
Not raw message throughput - that shards horizontally by conversation. The real constraint is the connection tier's memory footprint and the presence registry's write rate, since every heartbeat, connect, and disconnect is a registry write; at billions of users this pushes toward sharding the presence registry itself by user_id range and keeping per-connection memory (buffers, TLS state) as small as possible, since that's what caps how many sockets a single box can hold.
Post a Comment
Add