Ordered Logs & Sync Cursors

Foundations

A messaging system has to answer two questions: in what order did things happen, and what has each device already received? A durable append-only log answers the first: each new entry gets the next sequence number in its log and is acknowledged once stored. The log, not any connection, is the source of truth.

Delivery takes two paths. A live connection pushes new entries immediately when one exists, as an optimization. A cursor answers the second question: each client stores the highest sequence number it has applied, and after any disconnect it asks for everything after that cursor. A reply may repeat entries, so the client skips any sequence number at or below its cursor.

Take a log with entries 1, 2, and 3, where 2 and 3 were appended while a client was offline after applying 1. On reconnect it asks for entries after 1, receives 2 and 3 in order, applies them, and moves its cursor to 3. Asking again returns nothing, and a replayed 2 would be skipped.

• Requirements

  Each worked system starts from a scoped list of verbs and numbers.

  Open lesson

• Partition Key Selection

  Messages are partitioned by conversation and ordered by a sequence number.

  Open lesson

• Queues

  Slow work leaves the request path through a queue.

  Open lesson

Case Scenario

To see how a log and a cursor survive disconnects, consider 1:1 messaging in an app like WhatsApp: two people exchange text, an online recipient sees it immediately, an offline one sees it on the next connection, and history is kept. Each message appends to the conversation's log, and a reconnecting phone fetches everything after its cursor.

Ana may send while Bo is offline, and either phone can reconnect after missing several messages. A live connection says only where a device is now; it cannot prove what that device has already received or preserve a common order after a network change. Sending must not wait for Bo’s phone, and reconnecting must recover every missed message without gaps or duplicates.