Caching Isn't Just About Speed — Cache-Aside, TTL, Invalidation, Stampedes and Hot Keys Explained

System Design Series — Part 4 of 5

🎯 Practice This Before Your Next System Design Interview

Reading about cache invalidation, cache stampedes and hot keys is one thing. Explaining them out loud, under pressure, in a 45-minute interview is a completely different skill.

→ intervues.club — the AI voice interview platform for real technical interviews. → candidates.intervues.club — practise system design and backend interviews right now, speak your answer, get a report.

Practice the interview before the interview.

1. Where We Left Off

In Part 3 we established that the database is almost always the bottleneck, and we climbed a cost ladder to fix it: fix the queries, add indexes, cache, replicate, partition, shard. This series is part of how we build and explain systems at Intervues — and why we test candidates on saying it out loud, not just knowing it (pricing).

We skipped past step three quickly. Let's not.

Because most engineers describe caching in interviews with one sentence:

"We'd add Redis to make it faster."

That answer is not wrong. It's just shallow. And it misses the point of what caching actually does in a distributed system.

Caching is not primarily a latency optimization. It is a load-shedding mechanism.

Speed is the side effect people notice. Database load reduction is the reason caching exists in scalable system design.

This distinction matters, because it changes what you cache and how you reason about failure.

2. Why Caching Works

Caching works because of one property that almost every real workload has:

Access is not uniform. A small fraction of the data serves the overwhelming majority of requests.

This is the Pareto distribution, and it shows up everywhere:

If 95% of your reads target 0.01% of your data, then holding that 0.01% in fast memory eliminates 95% of your database read load.

The second reason caching works is the memory hierarchy. The cost of fetching data varies by orders of magnitude depending on where it lives:

Notice that Redis is not fast compared to RAM. It's fast compared to a database doing real work.

The latency difference is 16x. The CPU difference on the constrained resource is close to infinite, because the cache hit consumed zero database CPU.

That's the real win.

3. Cache Hit, Cache Miss, and Cache Hit Ratio

Three terms that everything else depends on.

The cache hit ratio is the single most important number in your caching layer:

Cache Hit Ratio = Cache Hits / (Cache Hits + Cache Misses)

Here's why it matters far more than people assume. Consider 10,000 requests per second against a database that can handle 2,000 queries per second:

The critical insight for interviews:

The relationship between hit ratio and database load is not linear in the way that matters. Going from 90% to 95% halves your database load. Going from 95% to 99% halves it again.

Equally important in reverse: a hit ratio dropping from 99% to 90% is a 10x increase in database traffic. That is how caches take down systems.

Your cache is not a speed feature. It is load-bearing infrastructure.

4. The Cache-Aside Pattern

The cache-aside pattern (also called lazy loading) is the default caching strategy in most systems. The application manages the cache explicitly.

Read path

Write path

Note the order: write to the database first, then delete from the cache. Reversing it opens a window where a concurrent read can repopulate the cache with the old value before the write lands.

Why cache-aside dominates:

• The cache only ever holds data someone actually asked for
• A cache failure is survivable — you fall through to the database
• It works with any datastore and requires no special integration
• It's simple to reason about

Its weaknesses:

• Every cache miss pays the full latency penalty
• A cold cache means the database absorbs everything
• The application is responsible for invalidation, which is where bugs live

Alternatives worth naming

In an interview, naming cache-aside and explaining why you'd pick it over write-through is worth more than listing all four.

5. TTL and Cache Invalidation

There's a well-worn joke that the two hard problems in computer science are cache invalidation and naming things. The joke persists because it's accurate.

Cache invalidation is hard because the cache has no way to know that the underlying data changed.

You have exactly three strategies.

TTL

TTL (time to live) is the expiry duration on a cache entry. It is the safety net that makes every other mistake survivable — if you forget an invalidation somewhere, TTL eventually fixes it.

Choosing TTL is a direct trade between freshness and load:

Practical TTL guidance by data type:

The rule to state in an interview:

TTL should be set by how much staleness the business can tolerate, not by how much staleness feels comfortable.

Explicit Invalidation

Invalidating product:42 is trivial. Remembering that this product also appears in six cached list views, three search result sets and a homepage block is where correctness bugs are born.

Mitigations:

• Key namespacing — product:42:* so you can invalidate a family
• Versioned keys — bump catalog:v7 to catalog:v8 and every derived key naturally misses
• Short TTLs on derived data — accept that aggregates are eventually correct
• Event-driven invalidation — publish a change event, let subscribers invalidate their own keys

Stale Data

Stale data is the price of admission for caching. It is not a bug to be eliminated; it is a trade-off to be scoped.

The same consistency trade-off we saw with read replicas in Part 3 applies here, just more aggressively — a cache is a replica with no replication protocol at all.

6. The Layers of Caching

Caching is not one thing in one place. In a mature system it is a chain, and each layer that absorbs a request removes load from every layer behind it.

Client-Side Caching

Client-side caching is the cheapest cache in existence, because the request never happens at all.

An ETag + 304 Not Modified response is often overlooked: the request happens, but the body doesn't transfer and your application does almost no work.

CDN Caching

CDN caching puts copies of your content in edge locations physically near users.

The CDN gives you two wins simultaneously: it eliminates geographic latency, and it absorbs the vast majority of requests for cacheable content before they ever touch your infrastructure.

Cache at the CDN: images, CSS, JS, fonts, video, public API responses, entire rendered pages for anonymous users.

Don't cache at the CDN: personalized responses, authenticated content, anything with a Set-Cookie.

Application Caching

Application caching is in-process memory inside your app server — a hash map, an LRU structure, a memoized function.

It is the fastest possible cache — nanoseconds, no network hop. Its limitation is that each instance has its own copy, so:

• Memory is duplicated N times
• Invalidation across instances is genuinely hard
• Instances can disagree with each other

Use it for small, slow-changing, non-critical data. Don't use it for user data.

Distributed Caching and Redis

Distributed caching puts the cache in a shared external service so every application instance sees the same data.

Redis is the default choice, and it's worth knowing why beyond "it's fast":

• Single-threaded command execution, so operations are atomic without locking
• Rich data structures — strings, hashes, sorted sets, sets, streams, HyperLogLog
• Native TTL per key
• Atomic primitives (SETNX, INCR) that make distributed locking and rate limiting easy
• Optional persistence, replication and cluster mode

The properties that matter architecturally:

That last point connects straight back to Part 2. Adding Redis doesn't eliminate the bottleneck — it moves it. You will eventually see Redis CPU at 95% while your database idles.

7. Cache Eviction and LRU

A cache has finite memory. When it fills, something must go. That's cache eviction.

LRU

LRU (Least Recently Used) evicts whatever hasn't been touched for the longest time. It's the default for good reason: it approximates "keep the hot data" without needing to know anything about your workload.

LFU (Least Frequently Used) is better when you have a stable hot set and occasional large scans — LRU can be polluted by a one-time bulk read that evicts your genuinely hot keys.

The policy to actually choose in Redis for a cache is allkeys-lru (or allkeys-lfu). The dangerous one is noeviction, which is the default in some configurations: instead of dropping cold keys, Redis starts rejecting writes, and your application starts throwing errors on a cache write path that you probably didn't wrap in a try/catch.

Cache Warming

Cache warming is pre-populating the cache before traffic arrives, so you never serve production traffic against a cold cache.

Ways to warm a cache:

• Run a background job that loads the known-hot key set before routing traffic
• Replay a sample of production read traffic against the new instance
• Use Redis persistence (RDB/AOF) so a restart restores the previous dataset
• Roll deploys gradually so a fraction of traffic naturally repopulates before full cutover

This is exactly why "just restart Redis" is a dangerous instinct during an incident. Flushing a cache under production load hands your database a 100% miss rate on the spot.

8. Cache Stampede and the Thundering Herd

This is the failure mode that separates people who have run caches in production from people who have read about them.

The Cache Stampede

A cache stampede happens when a popular key expires and hundreds or thousands of concurrent requests all miss simultaneously — and all of them independently decide to go rebuild it.

One key expiring generated 5,000 identical queries. The database was comfortably handling 100 QPS a second earlier.

This is the thundering herd problem: many processes waking simultaneously to do the same work.

Fix 1 — Locking / Single-Flight

Let exactly one request rebuild the value. Everyone else waits or serves stale.

SET key value NX EX 10 is atomic in Redis — this is a two-line fix that eliminates the entire class of problem.

Fix 2 — Probabilistic Early Expiry

Instead of all requests seeing expiry at the same instant, have requests probabilistically refresh the value as it approaches expiry.

Fix 3 — TTL Jitter

If you populate 10,000 keys in a warming job with identical TTLs, they all expire in the same second. Add randomness.

Fix 4 — Serve Stale While Revalidating

Store the value with a logical expiry that's shorter than the physical TTL. Past logical expiry, serve the old value immediately and refresh asynchronously.

The user gets a fast response with data that's a few seconds old. Your database gets one query instead of five thousand.

9. Hot Keys

A hot key is a single cache key receiving a disproportionate share of traffic. It's the caching-layer version of the hot shard from Part 3.

Redis Cluster shards by key hash. One key lives on exactly one node. If that key is hot enough, you cannot scale it by adding Redis nodes — the key doesn't move, and it can't split.

Where hot keys come from:

• A celebrity user's profile on a social app
• A single product during a flash sale
• The global feature-flag or config key that every request reads
• A leaderboard or homepage block read by every session
• A rate-limit counter for a shared resource

Mitigations:

The first one is usually enough. If every app server holds config:global in local memory for one second, a key receiving 500,000 requests/sec across 50 servers now receives 50 requests/sec at Redis.

That's a 10,000x load reduction for one second of staleness. Very often, a good trade.

10. Cache Failure and the Cache-Miss Storm

Here is the scenario that turns a cache from an optimization into a liability.

This is a cache-miss storm, and it is one of the most common causes of large-scale outages. The database was healthy. The application was healthy. A cache — a component added to make things better — took the system down.

The uncomfortable realization:

Once you depend on a 95% hit ratio to survive, your cache is no longer an optimization. It is a critical dependency, and you must engineer it like one.

Designing for cache failure

Point 7 deserves emphasis. If your Redis client has no timeout and Redis becomes slow rather than dead, every request now waits on Redis and then waits on the database. A degraded cache with no timeout is worse than no cache at all.

Rule: cache calls should have an aggressive timeout (tens of milliseconds), and a cache error should fall through to the database, not raise.

11. Consistency Trade-Offs

Every cache is a deliberate decision to serve possibly-wrong data in exchange for capacity.

The framework for deciding, which is what interviewers want to hear:

Concrete calls:

Notice the pattern: the same entity gets different treatment depending on what the read is for. Price on a listing page and price at checkout are the same column and completely different caching decisions.

12. Caching as a Database-Load Reduction Mechanism

Let's return to the thesis with numbers.

An e-commerce backend, following on from Part 3:

Now introduce caching layer by layer:

A 44x reduction in database load. Zero database changes. No sharding. No replicas.

This is why caching sits at step three of the cost ladder, above indexes and below replication:

Caching is the highest-leverage step on the entire ladder. It buys you an order of magnitude of capacity for days of work, and it buys you the time to do the harder steps properly.

But it buys that with staleness and operational risk. Those are the words that need to be in your answer.

13. What Interviewers Are Actually Listening For

When someone asks "how would you use caching here?", saying "add Redis" is a level-1 answer. Here's the shape of a level-5 answer:

Phrases that signal you've run this in production:

• "What's the read/write ratio on this entity, and how stale can it be?"
• "I'd use cache-aside — a cache failure degrades to the database rather than to an error."
• "I'd add jitter to the TTL so keys don't expire in lockstep."
• "That key is going to be hot — I'd put a one-second local cache in front of Redis."
• "If Redis goes down, we go from 500 QPS to 10,000 QPS on the database. I'd need a circuit breaker."
• "I'd delete the key after the database write, not before."
• "What's our target hit ratio? Below 90% the caching layer isn't earning its complexity."

The last one is the strongest signal available, because it treats caching as something to measure, not something to add.

14. Key Terms Recap

15. Final Takeaway

Caching is the highest-leverage tool in scalable system design, and the most commonly misunderstood.

It is not a speed feature. It is a mechanism for removing work from your constrained layer so that the system as a whole can handle more.

That reframing changes everything:

• You cache what is expensive and repeated, not what is merely slow
• You set TTL by staleness tolerance, not by intuition
• You measure hit ratio, because it maps directly to database load
• You plan for stampedes, hot keys and cache failure, because a cache your system depends on is critical infrastructure
• You accept staleness deliberately, and you can say exactly how much

The two sentences to carry into any system design interview:

A cache doesn't make your system faster. It makes your bottleneck do less work.

And the moment you depend on your cache, its failure mode becomes your system's failure mode.

Design it accordingly. Hiring teams who need this depth in screening can see structured voice interviews.

Next in the series — Part 5: Consistency, Availability and Designing for Failure.

🎯 Now Go Explain This Out Loud

You can read every article on caching strategies, cache invalidation, Redis and cache stampedes ever written and still freeze when an interviewer asks "what happens to your database when Redis goes down?"

The gap between knowing and explaining under pressure is where most technical interviews are lost.

→ candidates.intervues.club — start a real AI voice interview right now. Speak your system design answer out loud, get a structured report on your reasoning, clarity and depth. Free to start.

→ intervues.club — hiring? Run AI voice technical interviews at scale, with candidates who can actually explain their architecture decisions.

No AI copilot feeding you answers. Just you, the question, and your ability to reason.

Practice the interview before the interview. → candidates.intervues.club

Part 4 of 5 in the System Design Fundamentals series. Part 1: Load, Latency and Throughput. Part 2: Horizontal Scaling and Bottleneck Migration. Part 3: The Database Is the Bottleneck. Part 5: Consistency, Availability and Failure.

Tags: system design interview, caching strategies, cache-aside pattern, cache invalidation, cache hit ratio, TTL, Redis, distributed caching, CDN caching, cache stampede, thundering herd, hot key problem, LRU eviction, cache warming, cache-miss storm, eventual consistency, database load reduction, scalable system design, backend engineering, distributed systems