On this page
Description
The home feed is the heart of most social apps, and "design a news feed" is one of the most common system design questions. It looks simple - show recent posts from the people you follow - but at scale it forces nearly every big trade-off: doing work when data is written versus when it's read, handling a few users who are millions of times more popular than the rest, caching, queues, and media storage.
In this lesson:
- Why building the feed on every read stops working.
- The numbers, and what they tell you.
- Fan-out on write: building feeds ahead of time.
- The celebrity problem, and the hybrid fix.
- Inactive users, reading the feed, and storing photos.
The first version, and why it breaks
You're building the home feed for a social app: open it, and you see recent posts from the people you follow. The first version runs a query like this every time someone opens the app:
SELECT p.* FROM posts p
JOIN follows f ON f.followee_id = p.author_id
WHERE f.follower_id = :me
ORDER BY p.created_at DESC LIMIT 20;
That works fine for a few thousand users. Now picture someone who follows 300 accounts. Every time they open the app, the database has to dig through 300 people's posts, sort them and pick the newest 20. Multiply that by hundreds of millions of people opening the app ten times a day, and no database keeps up.
The way out is to stop building the feed when someone asks for it, and build it ahead of time instead.
The numbers
Write these down first. Every decision below follows from them.
- 300 million daily users × 10 feed opens = 3 billion reads a day, about 35,000 a second, peaking near 100,000 a second.
- 100 million new posts a day, about 1,200 a second.
- The average user has about 200 followers. A few accounts have tens of millions.
- A feed should load in under 300 ms, and a new post may take up to a minute to reach followers.
Reads outnumber writes about 30 to 1, and we're allowed a minute of lag. Both facts point the same way: do the expensive work when a post is written, so reading is cheap.
Fan-out on write
When someone posts, push the post's ID into the timeline of every follower. A timeline is a short, precomputed list per user, kept in a cache such as a Redis list: newest first, capped at a few hundred entries. Opening the feed becomes one cache read, plus fetching the posts by ID.
The pushing happens in the background, so the person posting doesn't wait for it:
The post service saves the post, publishes a "new post" event and returns right away. Fan-out workers read the event, page through the author's followers, and add the post ID to each follower's timeline. At 1,200 posts a second with 200 followers each, that's about 240,000 timeline writes a second. Spread across many workers and cache nodes, that's routine.
Timelines hold IDs, not copies of posts. If someone edits or deletes a post, you change one row, not millions of timeline entries. The cache stays small, too.
The celebrity problem
Now an account with 50 million followers posts. Fan-out on write turns that one post into 50 million timeline writes: minutes of work for the fan-out workers, repeated with every post, and mostly for followers who won't open the app today. Pushing doesn't work for them.
Fan-out on read does: don't push celebrity posts at all. When someone opens their feed, fetch the recent posts of the few celebrities they follow and merge them into their precomputed timeline. So real systems use a hybrid:
- Ordinary accounts: fan-out on write.
- Accounts above a follower threshold (say 1 million): fan-out on read.
Read more - where to put the threshold
The threshold is a cost trade-off, not a magic number. Pushing costs one timeline write per follower per post. Pulling costs one extra lookup per feed open for everyone who follows the account. An account with many followers who post rarely but read often is cheaper to pull; a small account is always cheaper to push. Teams tune the threshold by measuring both costs.
Inactive users
Most registered users don't open the app on a given day. Pushing posts into their timelines wastes most of those writes and a lot of cache memory. Skip users who haven't been active for a while (say 30 days), and rebuild their timeline from the posts database when they come back. It's a one-time cost when they log in, and nobody notices.
Reading the feed
- Get a page of post IDs from the user's timeline in the cache, using a cursor.
- Merge in recent posts from the celebrities they follow.
- Hydrate the posts: look them up by ID in a post cache, which sits in front of the database.
- Return the page.
If a timeline is missing (the cache evicted it, or a cache node failed), rebuild it from the database. The timeline cache is a speed-up, never the source of truth.
Photos and media
Photos are big blobs, and storing them in database rows makes the database huge and slow. Keep them in an object store such as Amazon S3, and keep only the object key in the database.
- Let the app upload directly to the object store with a pre-signed URL, so gigabytes never stream through your app servers.
- Serve photos through a CDN that uses the object store as its origin.
- Use versioned keys (
photos/42/v2.jpg), so the CDN can cache each version for a long time.
Which approach when
| Situation | Approach |
|---|---|
| Ordinary account posts | Fan-out on write into followers' timelines |
| Account with millions of followers posts | Fan-out on read: merge their posts in when a feed is opened |
| Follower inactive for 30+ days | Skip them; rebuild their timeline when they return |
| Timeline missing from the cache | Rebuild it from the posts database |
| A post is edited or deleted | Change the post row; timelines hold only IDs |
Common mistakes
- Building the feed with a join at read time. Fine for a prototype, hopeless at 100,000 reads a second.
- Fanning out synchronously. The person posting would wait for millions of writes. Use a queue.
- Pushing celebrity posts like everyone else's. One post becomes tens of millions of writes.
- Copying whole posts into timelines. Edits and deletes then touch millions of entries.
- Treating the timeline cache as the only copy. A cache node failure would then lose feeds.
Check yourself
Try to answer before opening each one - recalling beats re-reading.
1. Why do timelines store post IDs rather than copies of posts?
Show answer
An edit or deletion then updates one row instead of millions of timeline entries, and the timeline cache stays small. The cost is a hydration step on read, which a post cache makes cheap.
2. A user follows 3 celebrities and 297 ordinary accounts. How is their feed put together?
Show answer
The 297 ordinary accounts' posts are already in their precomputed timeline (fan-out on write). When they open the app, recent posts from the 3 celebrities are fetched and merged in (fan-out on read).
3. A timeline cache node fails. What happens to the users whose timelines lived there?
Show answer
Their timelines are missing, so their next feed read rebuilds them from the posts database. That read is slower, but it's correct. That's why the cache must never be the only copy of anything.
Key takeaways
- Build feeds ahead of time: fan-out on write into per-user timelines, in the background through a queue.
- Timelines hold post IDs. Hydrate them on read from a post cache.
- Pull celebrities' posts at read time instead of pushing them to millions.
- Skip inactive users, and rebuild their timelines when they come back.
- Photos live in object storage, are uploaded directly, and are served through a CDN.
- The timeline cache can always be rebuilt from the database.