Hot Shards

Overview

Hot shards are those unlucky partitions that see a disproportionate amount of traffic or work. Even if you use a “good” random hash function, hot shards still appear in production systems for reasons that have nothing to do with the hash function itself.

This post focuses on solutions: how real-world systems (DynamoDB, Spanner, Cassandra, ScyllaDB, Redis Cluster, Elasticsearch, etc.) actually mitigate hot shards.

TL;DR

• Random / uniform hashing only fixes key distribution problems.
• Hot shards still appear due to workload skew, time-series traffic, global keys, query patterns, and physical node imbalance.
• Production systems lean on microshards, adaptive capacity, key-splitting, write-sharding, replica-aware routing, and CRDT-style counters to keep things stable.

1. Recap: Why Hot Shards Exist Even With Random Hashing

Before talking about solutions, we need to be precise about what we’re solving.

A shard can be “hot” for several root causes:

1. Hot keys

   • A few keys receive 1000× more traffic than others (celebrity users, popular config keys, global counters).

2. Temporal / time-series hotspots

   • Recent data or monotonic IDs funnel all new writes into a small region.

3. Query skew & range scans

   • Queries that repeatedly target the same narrow slice of the key space, or scan wide ranges on a subset of shards.

4. Physical resource imbalance

   • One node has worse hardware, heavier compaction, worse cache locality, or is just “unlucky” with shard placement.

5. Global logical keys / coordination hotspots

   • Global locks, global version numbers, global rate-limiter buckets, etc., centralize traffic on a single shard.

6. Storage engine hotspots

   • LSM compaction, GC, or disk-level behavior cause a shard to be much more expensive to maintain than others.

Random hashing only addresses:

“Are keys evenly distributed across shards?”

It does not guarantee:

“Is work evenly distributed across shards?”

The rest of this post is about how to fix each root cause.

2. Design Principles: The “Anti–Hot-Shard” Mindset

The techniques differ, but the guiding principles are fairly consistent:

• Over-partition (many small shards, not a few large ones).
• Monitor traffic + CPU + storage per shard.
• Move or split work when it becomes skewed.
• Avoid single global choke points (keys, locks, counters).
• Push complexity to the edges (routers, read aggregators, materialized views) rather than the hottest shard.

You can think of the overall architecture as:

“Lots of tiny pieces that can be moved around, and enough observability to know what to move.”

We’ll now go cause-by-cause.

3. Hot Keys → Key-Level Load Splitting

3.1. Problem

A single key (or small set of keys) is much hotter than the rest:

• A celebrity’s user profile
• A single “global-progress” row
• A config document everyone reads
• A global counter or rate limiter bucket

Hashing doesn’t help: that one key maps to one shard.

3.2. Solution Pattern: “Key Salting” / “Key Splitting”

Instead of:

Key: user:12345

You create logical shards of the same key:

user:12345#0
user:12345#1
user:12345#2
...

Writes are spread across these “sub-keys”, and reads aggregate them.

3.3. Alternate Solution Pattern: Reverse Architecture of Writes and Reads

Twitter-style systems often avoid a hot shard on content poster by not using a single “global feed.” Instead, they either:

• Fan-out-on-write for normal keys: on tweet, push the tweet into each follower’s own timeline (sharded by follower ID), or
• Fan-out-on-read / hybrid for hot keys: for very high-follower accounts, don’t push to all followers at write time; instead, pull those tweets into a user’s home timeline dynamically when they fetch it. Tweet storage is sharded by tweet_id (high entropy), chunking follower lists, and caching tweet reads. Fan-out-on-read builds timelines from these sharded components — it doesn’t rely on key salting of the tweet itself.

3.4. Variants

3.5. When to use

• You see one customer, one tenant, or one resource dominating load.
• Per-key QPS is often more useful than per-shard QPS for triggering this.

4. Temporal / Time-Series Hotspots → Write Sharding

4.1. Problem

Time-series workloads tend to pile writes onto the “latest” region:

• Monotonic IDs (Snowflake, UUIDv7)
• Time-bucketed keys (events:2025-11-29-12:00)
• Logs / metrics / traces

Even if the hash function is “good”, the newest buckets or IDs very often land on a small cluster of shards.

4.2. Solution Pattern: “Write Sharding” for Time Buckets

You keep the time locality at a coarse level (day/hour) but add randomness inside the bucket:

events:2025-11-29T12:00#0
events:2025-11-29T12:00#1
...
events:2025-11-29T12:00#31

New writes for “now” are scattered across multiple sub-buckets. Reads that look at “12:00” will fan out across these 32 sub-buckets.

At 1:00PM, there would need to be a re-allocation of shards.

4.3. Variants

5. Query Skew & Range Scans → Smarter Query Routing and Index Sharding

5.1. Problem

Even when keys are evenly distributed, queries might not be:

• “Give me last 100 events for user X” → 1 shard, very often
• “Scan orders between ID A and ID B” → specific small range
• “Search posts for tag = hot” → only a few shards have relevant data

Hashing doesn’t help here because the skew comes from the query pattern, not key distribution.

5.2. Solution Pattern: Distributed Indexes + Smart Routing

You want the router/query layer to know:

• Which shards have relevant data
• How to spread load across replicas
• When to offload heavy aggregations to background jobs/materialized views

5.3. Techniques

6. Physical Node Imbalance → Microshards & Automatic Rebalancing

6.1. Problem

Even with a perfect logical distribution, one machine can be overloaded:

• Noisy neighbor on shared hardware
• Hotter set of partitions “by accident”
• More compaction / GC on a subset of shards
• One bad disk or NIC

Now you have a hot shard by virtue of placement, not data distribution.

6.2. Solution Pattern: Many Small Virtual Shards (Microshards)

Instead of having, say, 64 partitions each pinned to a single node, you use thousands of tiny logical partitions (virtual nodes, vnodes, microshards) spread over the cluster.

When a few microshards become hot, the cluster can:

• Move just those microshards to underutilized nodes
• Replicate them more aggressively
• Throttle or reshard them

6.3. Techniques

7. Global Logical Keys → Distributed State, Not Centralized State

7.1. Problem

Some things are inherently global:

• Global “current version” of a config
• Global counters (total users, total events)
• Global locks or leases
• Global rate limiter bucket

If you implement these as a single key or row, that key becomes a guaranteed hot spot.

7.2. Solution Pattern: Decompose Global State

Break global state into many smaller, less synchronized pieces:

• Sharded counters / CRDTs instead of a single counter.
• Per-scope limits instead of one global rate bucket.
• Distributed lock services instead of one database row.

7.3. Patterns & Tradeoffs

8. Storage-Engine Hotspots → Engine-Aware Design

8.1. Problem

Even if traffic is balanced, the underlying storage engine can make some shards more expensive to maintain:

• Compaction storms for specific SSTables/segments
• Write amplification skewed to a few partitions
• GC or memory pressure isolated to particular key ranges

8.2. Solution Pattern: Align Logical Sharding With Engine Behavior

Key ideas:

• Prefer append-friendly patterns over in-place updates.
• Use tiered storage (hot vs cold) so old data doesn’t penalize hot keys.
• Keep shard sizes bounded, so compaction jobs don’t balloon.

8.3. Techniques

9. Unified View: Root Causes vs. Solution Patterns

Here’s the “one big table” you can reference later.

10. Putting It All Together

If you were designing a new system today and wanted to be reasonably robust to hot shards, a good baseline would be:

1. Random hashing with many microshards

   • Start with 10×–100× more logical shards than physical nodes.
   • Route via consistent hashing.

2. Shard-aware router with rich telemetry

   • Track QPS, latency, CPU, and storage per shard.
   • Use this to drive resharding and autoscaling.

3. Hot-key detection + key-splitting

   • If per-key QPS crosses a threshold, automatically salt/split.
   • Expose a simple aggregation layer for reads.

4. Write sharding for time-series workloads

   • Use time buckets + randomness rather than purely monotonic IDs.
   • Keep bucket sizes bounded and split when needed.

5. Distributed index and query layer

   • Secondary indexes are themselves distributed.
   • Support scatter-gather with pruning and replica-aware routing.

6. Storage-engine-aware policies

   • Avoid huge shards; cap shard size.
   • Tune compaction to avoid IO storms.

7. Avoid single global keys

   • Use CRDT-style counters and hierarchical limits.
   • Reserve a dedicated, low-QPS system for locks and coordination.

That combination is more or less what the big players converge on, with different flavors and tradeoffs.