Sharding Strategies
Split data across machines by a chosen key — range, hash, geographic, or consistent hashing — to scale writes beyond one node.
What it is
Sharding partitions data across multiple nodes so each node owns a subset. The shard key determines placement. Range sharding (ranges of key values) preserves locality for range queries but risks hot shards at the ends. Hash sharding spreads writes evenly but kills range queries. Geographic sharding co-locates data with users for latency. Consistent hashing minimizes rebalancing when nodes are added or removed. The choice of shard key is the most consequential design decision in a sharded system — it's nearly impossible to change later.
Why senior interviewers ask
Sharding is the canonical 'walk me through your design' question. Picking the wrong shard key causes hot partitions, cross-shard joins, and rebalancing nightmares. Senior interviewers want to hear access patterns drive the choice.
Key points
- Range sharding: contiguous ranges per shard. Good for range scans, bad for time-ordered writes (latest range is hot).
- Hash sharding: hash(key) mod N. Even distribution, but range queries become scatter-gather.
- Consistent hashing: hash ring with virtual nodes — adding a node moves ~1/N of data instead of all of it.
- Geographic sharding: shard by region for latency and data residency (GDPR).
- Shard key must be high-cardinality and well-distributed to avoid hot shards.
- Cross-shard transactions are expensive (2PC, Two-Phase Commit) or impossible — design to avoid them.
- Resharding is painful — pick the key you'll still want in 3 years.
- Hot shard mitigation: composite keys, salting, write fanout.
Pros
- Horizontal write scalability — add shards, throughput grows.
- Smaller per-node working set fits in cache better.
- Failure isolation — one shard down doesn't take out the whole system.
- Consistent hashing minimizes rebalancing cost.
Cons
- Cross-shard joins and transactions are slow or impossible.
- Hot shards from bad key choice are catastrophic.
- Operational complexity: rebalancing, backups, schema changes per shard.
- Resharding requires careful, often manual work.
When to choose
- Range sharding: range-scan-heavy workloads (time-bucketed analytics) where you can avoid hot ranges.
- Hash sharding: uniform write patterns, no range queries needed.
- Consistent hashing: cache layers (Memcached, Redis Cluster) where nodes come and go.
- Geographic sharding: global apps with data-residency requirements.
When to avoid
- Single-node throughput is enough — sharding adds operational tax you don't need yet.
- Workloads dominated by cross-entity transactions or joins.
Real systems
Interview probe
'You're sharding users by user_id hash. A user has 10M posts. How do you query their feed?' Answer: either co-locate posts with user (compound shard key user_id+post_id) or accept scatter-gather. Mention the tradeoff explicitly — that's the senior signal.