Distributed Caching in System Design
A cache inside one application instance works until the application scales out. With ten instances, each keeps its own copy, hit rates drop, memory is wasted, and an update in one instance leaves stale data in the other nine. A distributed cache solves this by moving cached data into a shared cluster that every instance uses.
This tutorial explains how a distributed cache spreads and protects data, the main topologies, how Redis Cluster places keys, and the problems that appear only at scale, such as hot keys and cache stampedes. It builds on What Is Caching and Cache Eviction Policies.
1. What Is a Distributed Cache?
A distributed cache is a cache whose data is stored across multiple networked servers and shared by many application instances. It appears to applications as one logical key-value store, while internally it partitions keys across nodes and usually replicates them for availability.
Common examples are Redis Cluster, Memcached, Hazelcast and Apache Ignite, as well as managed services such as Amazon ElastiCache and Azure Cache for Redis. They keep data in memory, so reads typically take well under a millisecond inside a data centre.
2. Distributed Cache Architecture
Applications never need to know which node holds a key. The cache client, or a proxy in front of the cluster, computes the owner from the key and sends the request there.
3. How a Distributed Cache Handles a Request
Take a product page served by any one of many application instances:
- Locate: the cache client hashes the key
product:42to find the node that owns it. - Read: it sends
GET product:42to that node. On a hit, the value returns in well under a millisecond. - Miss: on a miss, the application loads the product from the database and writes it back with a TTL, so every other instance now finds it in the cache.
- Replicate: the owning node copies the change to its replica, which takes over if the node fails.
- Invalidate: when the product changes, the service deletes or updates the cached key, and the change is visible to all instances at once.
4. Distributed Cache Topologies and Key Concepts
Partitioned (sharded) cache
Keys are divided across nodes by hashing, so each node stores only part of the data. Adding nodes increases both memory and throughput. Clients use consistent hashing or fixed hash slots so that adding a node moves only a fraction of the keys.
Replicated cache
Every node holds the full data set. Reads are fast and highly available, but memory does not grow with the cluster and every write must be applied everywhere. It suits small, read-heavy reference data such as country lists or feature flags.
Near cache (two-level cache)
An in-process cache such as Caffeine sits in front of the shared cache. The hottest keys are answered without a network hop, while the shared cache provides consistency across instances. Invalidation messages, for example through Redis Pub/Sub, keep the local copies fresh.
Redis Cluster hash slots
Redis Cluster does not use a hash ring. It maps every key to one of 16,384 hash slots and assigns slot ranges to primary nodes, each of which can have replicas. Keys that must be used together in one command can share a slot through hash tags, such as {user:7}:cart and {user:7}:profile.
Problems that appear at scale
- Hot keys: one very popular key overloads the single node that owns it. Fix it with a near cache or by splitting the key into copies.
- Cache stampede: when a popular key expires, many requests miss at once and hit the database together. Use request coalescing, a short lock, or early refresh before expiry.
- Stale data: updates to the database must be followed by cache invalidation, or readers see old values until the TTL ends.
- Network partitions: a split cluster may serve inconsistent data or reject writes, depending on its settings.
5. Advantages and Disadvantages
Advantages
- One shared copy of cached data for every application instance.
- Memory and throughput scale by adding nodes.
- Replicas survive node failures without a cold cache.
- Application instances stay stateless and easy to scale.
Disadvantages
- Every access is a network call, slower than an in-process cache.
- One more distributed system to operate, secure and monitor.
- Consistency between cache and database must be managed carefully.
- Multi-key operations are limited when keys live on different nodes.
6. Local vs Distributed Cache
| Aspect | Local (in-process) cache | Distributed cache |
|---|---|---|
| Location | Inside each application instance | Separate cluster shared by all instances |
| Latency | Nanoseconds to microseconds | Sub-millisecond, plus network |
| Consistency across instances | Each instance may differ | One shared view |
| Capacity | Limited to one process's memory | Grows with the cluster |
| Survives application restart | No | Yes |
| Examples | Caffeine, Ehcache (local mode) | Redis Cluster, Memcached, Hazelcast |
7. Real-World Use Cases
- Session storage: any instance can serve any user because sessions live in the shared cache.
- API response and query caching: expensive results are computed once and reused by every instance.
- Rate limiting: counters in a shared cache enforce limits across all instances; see Rate Limiting.
- Leaderboards and counters: sorted sets and atomic increments in Redis.
- Microservices: each service caches the data it owns, with keys prefixed by service name to avoid clashes.
8. Best Practices
- Design keys deliberately: use clear, versioned names such as
product:v2:42, and hash tags only where needed. - Set TTLs with jitter to avoid synchronized expiry and stampedes.
- Plan for cache failure: the application must keep working, more slowly, when the cache is down; use timeouts and circuit breakers.
- Replicate across availability zones and enable automatic failover.
- Secure the cluster: keep it on a private network, require authentication and use TLS.
- Watch hit ratio, latency, memory and evictions per node to spot hot keys and imbalance.
- Keep values small: large values slow the network and block other requests on the same node.
9. Summary
- A distributed cache stores cached data across a cluster shared by all application instances.
- Partitioned caches scale capacity; replicated caches maximize read availability; near caches cut latency for hot keys.
- Redis Cluster maps keys to 16,384 hash slots owned by primaries with replicas.
- At scale, watch for hot keys, cache stampedes and stale data.
- Treat the cache as an optimization: the system must still work when it is unavailable.
Next, learn how applications reuse database connections efficiently in Database Connection Pooling.
Distributed Caching: Top Interview Questions
These are the distributed caching questions most often asked in system design interviews, with short answers you can explain in your own words.
Q1. Why use a distributed cache instead of a local cache?
With many application instances, local caches duplicate data, give each instance a different view, and lose everything on restart. A distributed cache gives all instances one shared, consistent copy that scales with the cluster and survives application restarts.
Q2. How does Redis Cluster decide which node stores a key?
It computes CRC16(key) mod 16384 to get a hash slot, and each primary node owns a range of the 16,384 slots. Hash tags in curly braces make related keys land in the same slot.
Q3. How can you stop a cache stampede from overloading the database?
When a popular key expires, many concurrent requests miss and all query the database at once. Prevent it by letting only one request rebuild the value (a lock or request coalescing), refreshing the value shortly before it expires, and adding random jitter to TTLs.
Q4. How do you handle a hot key in a distributed cache?
A hot key overloads the one node that owns it. Options include a near cache in the application, read replicas for that key, or splitting it into several copies with suffixes and reading a random copy.
Q5. What should happen when the distributed cache is down?
The application should fall back to the database with timeouts and a circuit breaker, possibly with rate limiting to protect the database. The cache must be an optimization, not a dependency that takes the whole system down.