Sharding by Access Key

Foundations

So far, one primary held all the data, with copies for reads. Now, what if writes or storage outgrow any single machine, so copies cannot help and the data itself must be split?

Sharding splits rows across several primaries by a shard key. Count first: divide the write rate by one machine's write limit and the storage by one machine's disk, round both up, and take the larger. Then choose the key the hottest query already carries and hash it to a shard, so that query touches one shard. Queries without that key must ask every shard or use a separate index.

Take 12,000 writes per second against a 5,000-per-second limit per machine, and 18 TB against 4 TB disks. Writes need 3 shards and storage needs 5, so storage sets the count: 5 shards of about 2,400 writes per second and 3.6 TB each. Keyed by id, a lookup by id hashes straight to one shard.

• Databases

  Queues and partitions sit around a primary database. That lesson covers the key and the index.

  Open lesson

Case Scenario

To see how a shard count and key follow from the numbers, consider a URL shortener like bit.ly grown to 20,000 creates per second and 30 TB: the primary's write throughput and disk are the wall, and reads are already cached. Eight shards keyed by short code keep every redirect a single-shard lookup.

The shortener’s primary has grown to 20,000 creates per second and 30 TB, beyond one machine’s comfortable write and disk limits. Reads are already cached, so adding read capacity does not remove either wall, while a poor data split could make every redirect search many places. The larger layout must preserve redirect as a single-key lookup.