Sharding: choosing the key you will live with
Sharding splits one dataset across machines so that each holds a slice, and almost every consequence (which queries stay cheap, which become a fan-out to every shard, where the hot spots form) follows from one choice, the shard key, which is also the hardest choice in the system to reverse.
What you'll be able to do
- Decide whether a system needs sharding at all, from storage and write numbers rather than from scale anxiety
- Compare range, hash and directory sharding on hot spots, range queries and rebalancing cost
- Pick a shard key from the dominant query, and measure the skew a bad key produces
- Price a cross-shard query, a global secondary index and a cross-shard transaction, and say which ones to design away
Before this: back-of-the-envelope, picking-a-datastore, replication-and-consistency
Log in to read the rest of this lesson
Sharding splits one dataset across machines so that each holds a slice, and almost every consequence (which queries stay cheap, which become a fan-out to every shard, where the hot spots form) follows from one choice, the shard key, which is also the hardest choice in the system to reverse. This walkthrough runs about 23 minutes, with runnable code you can edit and re-run as you read. A new account gets 7 days of Pro free, no card required.
Log in to continueThe first module of every track is free to read on the web: see what's open in System design.