Hybrid Fanout for Celebrities

Foundations

So far, one model served every source: pull merged at read time, and push copied at write time. Now, what if a few sources have millions of followers, so pushing their items swamps the workers, while pulling everything makes every read slow?

Use both, split by audience size. Below a follower threshold, chosen from worker capacity and the freshness target, items are pushed into timelines as before. Above it, an item is written once, to the source's own recent list. A read loads the reader's pushed timeline plus the recent lists of the few huge sources they follow, and merges them by time.

Take a threshold of 100,000 followers. An item from a source with 300 followers is pushed: 300 inserts. An item from a source with 10 million followers is stored once instead of 10 million times. A reader who follows two such sources pays two extra cached reads per view, plus a small merge.

• Hot-Key Mitigation

  The celebrity path is the hot-write fix from that lesson.

  Open lesson

Case Scenario

To see how hybrid fanout tames huge audiences, consider the celebrity path in a feed like X's: one account has 20 million followers, and pushing each of its posts into every timeline is the outage. Its posts are stored once in a cached recent list, and each follower's home merges them with the pushed timeline by time.

The background work that is harmless for an author with 200 followers becomes an outage for an account with 20 million. One post can occupy workers long enough to delay ordinary posts, while some readers follow dozens of such accounts and still expect correctly ordered home pages. One celebrity post must not create 20 million timeline rows.