Group Fanout with Pub/Sub & Read Cursors

Foundations

So far, a conversation had two members. Now, what if one message must reach hundreds of members, some online and some not, without the sender doing hundreds of writes?

Append the message once to the group's log, then publish it on a channel named for the group. Publish/subscribe, pub/sub, delivers it to every server currently holding a member's connection, and each server pushes it to its local members. History needs no copies: each member keeps a read cursor into the shared log, and the unread count is the log's length minus that cursor.

Take a group of 500 members where 120 are online across 8 servers. A new message costs 1 append and 1 publish, and the 8 servers push it to their 120 connected members. The other 380 read it later from their cursors. Copying into 500 inboxes would have cost 500 writes.

• Partition Key Selection

  Messages are partitioned by conversation and ordered by a sequence number.

  Open lesson

Case Scenario

To see how one append and a channel serve a whole group, consider group chat in an app like Slack: up to 500 members share a conversation, a message reaches online members quickly, and it appears in history for everyone. Each message is appended once to the group's log and published once to the servers holding members' connections.

A message in group g7 must reach online members quickly and remain in history for all 500 members. Making the sender touch every inbox or socket turns one post into 500 synchronous actions, and one busy room can pin the app server that accepted it. The sender’s request must do constant durable work and must not open 500 member connections.