On this page
Description
A chat app has to feel instant: you send "Lunch?", and it appears on your friend's phone before you've put yours down. It also has to keep every message in order on every device, never lose one it said was sent, and cope when phones drop off the network. Designing WhatsApp, Messenger or Slack is a classic interview question because those few requirements force a design quite unlike an ordinary web API: long-lived connections, servers that hold state, and per-conversation ordering.
In this lesson:
- Why polling fails, and the numbers to design for.
- Persistent connections (WebSockets), and why they make chat servers stateful.
- Routing a message to whichever server holds the recipient.
- Keeping messages in order and making retries safe.
- Storing history, read receipts, groups and presence.
The first version, and why it breaks
The first version of a messaging app usually polls: every two seconds, each app asks the server "anything new for me?" With 100 million people online, that's 50 million requests a second, and almost all of them come back empty. Worse, a message can still take up to two seconds to show up.
Chat feels instant because the server pushes a message the moment it exists. To push, the server needs an open connection to every online phone and laptop. That one requirement shapes the whole design.
The numbers
- 100 million daily users, with about 20 million connected at the busiest moment.
- 10 billion messages a day, about 115,000 a second, peaking near 300,000 a second.
- Delivery to an online recipient in under 500 ms. A message the server has acknowledged is never lost. History shows up on every device.
- One-to-one chats, and groups of up to 500 people.
Persistent connections
Each online app opens a WebSocket, a long-lived connection that both sides can send on, to a chat server. A chat server can hold on the order of 100,000 mostly quiet connections, so 20 million online users need a couple of hundred servers. The load balancer in front spreads connections across servers (least connections works well), rather than individual requests.
Here's the catch: these servers are stateful. Bob can only be reached through the one server that holds his connection. Any server that wants to deliver a message to Bob has to find that server first.
Routing a message
Alice is connected to chat server 3 and sends Bob a message:
- Server 3 receives the message on Alice's connection.
- It stores the message first, then tells Alice it's sent. Storing first is what makes "never lost" true.
- It asks the session registry where Bob is. The registry is a fast store, such as Redis, that maps each user to the server holding their connection. A server writes an entry when someone connects, and keeps it alive with heartbeats.
- If Bob is on server 17, server 3 forwards the message there (directly, or through a publish/subscribe channel per server), and server 17 pushes it down Bob's connection.
- If Bob isn't connected anywhere, the message goes to an offline queue. A worker sends him a push notification through Apple's or Google's push service (APNs or FCM). When Bob opens the app, it fetches everything after the last message it saw.
Order and duplicates
Two problems show up at scale.
Order. Messages from different people arrive at different servers, and phones' clocks don't agree. If you sorted by the sender's timestamp, two devices could show a conversation in different orders. The fix is a sequence number per conversation, handed out by the server when it stores the message. Every device shows the conversation in sequence order. You don't need one global counter for the whole system, which would be a bottleneck. Order only matters within a conversation.
Duplicates. A phone on a shaky train connection sends a message, doesn't hear back, and sends it again. The app gives every message a unique message ID, and the server checks it. If it's already stored, the server just says "sent" again instead of posting it twice.
Read more - why the app, not the server, picks the message ID
The ID has to exist before the first attempt leaves the phone, so the retry carries the same one. If the server assigned IDs, a retry would look like a brand-new message. A random UUID generated on the device works well.
Storing messages
Messages are written all the time and read one conversation at a time, newest first. A wide-column store (such as Cassandra) partitioned by conversation ID and sorted by sequence number fits that exactly: reading recent history is a single range scan of one partition. Old messages can move to cheaper storage.
Read receipts ("delivered", "read") are not flags on each message. In a 500-person group, that would be 500 flags per message. Instead, store one marker per person per conversation: "Bob has read up to sequence 1093."
Groups
A message to a 500-person group is stored once, under the group's conversation. Then it's delivered to each member: look up which servers members are connected to, send one batch to each of those servers, and queue push notifications for members who are offline. Huge broadcast channels, with tens of thousands of followers, work better with a pull model, just like celebrity posts in a news feed.
Presence
"Online" and "last seen" come from the registry's heartbeats. Telling every contact about every change is expensive: someone with 5,000 contacts whose phone keeps dropping signal on a train would cause a storm of updates. So presence updates are batched and slowed down, or only fetched when someone opens a conversation.
Which piece does what
| Need | Piece |
|---|---|
| Instant delivery to online users | WebSocket connections to chat servers |
| Find the server holding a user's connection | Session registry (user → server, kept alive by heartbeats) |
| Same order on every device | A sequence number per conversation, assigned when stored |
| Safe retries | A message ID from the app, checked by the server |
| History by conversation | A store partitioned by conversation ID, sorted by sequence |
| Offline users | An offline queue and push notifications |
Common mistakes
- Polling for new messages. Millions of empty requests a second, and still slow.
- Acknowledging before storing. If the server crashes in between, an "ack'd" message is lost.
- Ordering by the sender's clock. Devices disagree, so conversations look different on different phones.
- A flag per message per reader for receipts. Groups make this explode. Store "read up to" markers.
- Forgetting that chat servers are stateful. Without a registry, nobody knows where to deliver.
Check yourself
Try to answer before opening each one - recalling beats re-reading.
1. Why do chat servers need a session registry when ordinary web servers don't?
Show answer
A chat server holds live connections, so only the server with the recipient's connection can deliver to them. The registry maps users to servers, so any server can route a message to the right one. Ordinary web servers are stateless and can handle any request, so nothing needs to know where a user is.
2. Why a sequence number per conversation instead of the sender's timestamp?
Show answer
Device clocks disagree and can be wrong, so timestamps can put messages in different orders on different devices. A sequence number assigned by the server when it stores the message gives every device the same order.
3. An app resends a message after a timeout, but the first attempt actually worked. How does the design avoid a duplicate?
Show answer
The app sends the same message ID both times. The server sees it's already stored and simply acknowledges it again.
Key takeaways
- Push over persistent connections (WebSockets). Polling at this scale means millions of empty requests.
- Chat servers are stateful. A session registry maps each user to their server, so messages can be routed.
- Store before acknowledging. Sequence numbers per conversation fix the order; message IDs make retries safe.
- Partition messages by conversation. Read receipts are per-person "read up to" markers.
- Store group messages once, deliver per server, and push to offline members through a queue.
- Batch and slow down presence updates.