00:00

Distributed Caching in System Design

Last updated:

A cache inside one application instance works until the application scales out. With ten instances, each keeps its own copy, hit rates drop, memory is wasted, and an update in one instance leaves stale data in the other nine. A distributed cache solves this by moving cached data into a shared cluster that every instance uses.

This tutorial explains how a distributed cache spreads and protects data, the main topologies, how Redis Cluster places keys, and the problems that appear only at scale, such as hot keys and cache stampedes. It builds on What Is Caching and Cache Eviction Policies.

1. What Is a Distributed Cache?

A distributed cache is a cache whose data is stored across multiple networked servers and shared by many application instances. It appears to applications as one logical key-value store, while internally it partitions keys across nodes and usually replicates them for availability.

Common examples are Redis Cluster, Memcached, Hazelcast and Apache Ignite, as well as managed services such as Amazon ElastiCache and Azure Cache for Redis. They keep data in memory, so reads typically take well under a millisecond inside a data centre.

2. Distributed Cache Architecture

Distributed cache architecture Three application instances share one distributed cache made of three nodes. The cache client in each application hashes the key to find the node that owns it: node 1, node 2 or node 3, each holding about a third of the keys and each with a replica. On a cache miss the application reads from the database and writes the value into the cache. App instance 1cache client App instance 2cache client App instance 3cache client DISTRIBUTED CACHE CLUSTER Node 1~1/3 of keys replica Node 2~1/3 of keys replica Node 3~1/3 of keys replica Databaseon cache miss
Every application instance sees the same cache. A hash of the key decides which node owns it, replicas protect against node failure, and the database is used only on a miss.

Applications never need to know which node holds a key. The cache client, or a proxy in front of the cluster, computes the owner from the key and sends the request there.

3. How a Distributed Cache Handles a Request

Take a product page served by any one of many application instances:

  1. Locate: the cache client hashes the key product:42 to find the node that owns it.
  2. Read: it sends GET product:42 to that node. On a hit, the value returns in well under a millisecond.
  3. Miss: on a miss, the application loads the product from the database and writes it back with a TTL, so every other instance now finds it in the cache.
  4. Replicate: the owning node copies the change to its replica, which takes over if the node fails.
  5. Invalidate: when the product changes, the service deletes or updates the cached key, and the change is visible to all instances at once.

4. Distributed Cache Topologies and Key Concepts

Partitioned (sharded) cache

Keys are divided across nodes by hashing, so each node stores only part of the data. Adding nodes increases both memory and throughput. Clients use consistent hashing or fixed hash slots so that adding a node moves only a fraction of the keys.

Partitioned distributed cache Keys are spread across three nodes by hashing. user:1 and cart:9 live on node 1, product:42 lives on node 2, and session:7 lives on node 3. Each key is stored on exactly one primary node, so total capacity grows as nodes are added. hash(key)picks a node Node 1user:1 · cart:9 Node 2product:42 Node 3session:7 Total capacity = sum of all nodes
Each key lives on one node, chosen by hashing the key. Capacity and throughput grow with every node added, but a node failure loses its share unless it has replicas.

Replicated cache

Every node holds the full data set. Reads are fast and highly available, but memory does not grow with the cluster and every write must be applied everywhere. It suits small, read-heavy reference data such as country lists or feature flags.

Replicated distributed cache A write of product:42 goes to node 1 and is copied to node 2 and node 3, so every node holds the full data set and any node can serve a read. Writeproduct:42 Node 1all keys Node 2all keys Node 3all keys Total capacity = one node
Every node holds a full copy. Reads can go to any node and survive node failures, but capacity is limited to one node's memory and every write must reach all nodes.

Near cache (two-level cache)

An in-process cache such as Caffeine sits in front of the shared cache. The hottest keys are answered without a network hop, while the shared cache provides consistency across instances. Invalidation messages, for example through Redis Pub/Sub, keep the local copies fresh.

Near cache in front of a distributed cache An application instance first checks a small in-process near cache. On a miss it asks the shared distributed cache, and on a miss there it reads the database. Changes are announced through invalidation messages so near caches drop stale copies. APP INSTANCE Near cache (L1)in-process Shared cache (L2)Redis cluster Database miss miss invalidate on change
A small in-process cache (L1) answers the hottest keys in nanoseconds; the shared cache (L2) and the database are consulted only on misses. Invalidation messages keep L1 copies from going stale.

Redis Cluster hash slots

Redis Cluster does not use a hash ring. It maps every key to one of 16,384 hash slots and assigns slot ranges to primary nodes, each of which can have replicas. Keys that must be used together in one command can share a slot through hash tags, such as {user:7}:cart and {user:7}:profile.

Redis Cluster hash slots Redis Cluster divides the key space into 16,384 hash slots. With three primaries, the default layout gives slots 0 to 5460 to primary A, 5461 to 10922 to primary B and 10923 to 16383 to primary C. The key product:42 is hashed with CRC16 modulo 16384 to find its slot. CRC16(key)mod 16384 → slot Primary Aslots0 – 5460 replica Primary Bslots5461 – 10922 replica Primary Cslots10923 – 16383 replica
Redis Cluster maps every key to one of 16,384 slots with CRC16(key) mod 16384, and each primary owns a range of slots. Adding a node moves whole slots, not individual keys.

Problems that appear at scale

  • Hot keys: one very popular key overloads the single node that owns it. Fix it with a near cache or by splitting the key into copies.
  • Cache stampede: when a popular key expires, many requests miss at once and hit the database together. Use request coalescing, a short lock, or early refresh before expiry.
  • Stale data: updates to the database must be followed by cache invalidation, or readers see old values until the TTL ends.
  • Network partitions: a split cluster may serve inconsistent data or reject writes, depending on its settings.

5. Advantages and Disadvantages

Advantages

  • One shared copy of cached data for every application instance.
  • Memory and throughput scale by adding nodes.
  • Replicas survive node failures without a cold cache.
  • Application instances stay stateless and easy to scale.

Disadvantages

  • Every access is a network call, slower than an in-process cache.
  • One more distributed system to operate, secure and monitor.
  • Consistency between cache and database must be managed carefully.
  • Multi-key operations are limited when keys live on different nodes.

6. Local vs Distributed Cache

AspectLocal (in-process) cacheDistributed cache
LocationInside each application instanceSeparate cluster shared by all instances
LatencyNanoseconds to microsecondsSub-millisecond, plus network
Consistency across instancesEach instance may differOne shared view
CapacityLimited to one process's memoryGrows with the cluster
Survives application restartNoYes
ExamplesCaffeine, Ehcache (local mode)Redis Cluster, Memcached, Hazelcast

7. Real-World Use Cases

  • Session storage: any instance can serve any user because sessions live in the shared cache.
  • API response and query caching: expensive results are computed once and reused by every instance.
  • Rate limiting: counters in a shared cache enforce limits across all instances; see Rate Limiting.
  • Leaderboards and counters: sorted sets and atomic increments in Redis.
  • Microservices: each service caches the data it owns, with keys prefixed by service name to avoid clashes.

8. Best Practices

  • Design keys deliberately: use clear, versioned names such as product:v2:42, and hash tags only where needed.
  • Set TTLs with jitter to avoid synchronized expiry and stampedes.
  • Plan for cache failure: the application must keep working, more slowly, when the cache is down; use timeouts and circuit breakers.
  • Replicate across availability zones and enable automatic failover.
  • Secure the cluster: keep it on a private network, require authentication and use TLS.
  • Watch hit ratio, latency, memory and evictions per node to spot hot keys and imbalance.
  • Keep values small: large values slow the network and block other requests on the same node.

9. Summary

  • A distributed cache stores cached data across a cluster shared by all application instances.
  • Partitioned caches scale capacity; replicated caches maximize read availability; near caches cut latency for hot keys.
  • Redis Cluster maps keys to 16,384 hash slots owned by primaries with replicas.
  • At scale, watch for hot keys, cache stampedes and stale data.
  • Treat the cache as an optimization: the system must still work when it is unavailable.

Next, learn how applications reuse database connections efficiently in Database Connection Pooling.