Scenario: The Midnight Cache Expiration
A social media platform caches the profile data of its most famous celebrity in a Redis cluster to protect the primary SQL database. The cache is configured with a strict TTL (Time To Live) that expires exactly at midnight. The celebrity’s profile receives roughly 10,000 views per second.
Q:The clock strikes 12:00:00, and the Redis key expires. At 12:00:01, the database server instantly catches fire and crashes. What specific concurrency phenomenon just took down the system? Reveal â–¾
This is a Cache Stampede (also known as the Thundering Herd problem).
At exactly midnight, the cache is invalidated. Over the next few milliseconds, 10,000 concurrent HTTP requests arrive at the application servers. Every single thread queries the cache, receives a “Cache Miss,” and decides it must be the one to fetch the data from the primary database to repopulate the cache. The SQL database, normally shielded by Redis, is suddenly slammed by 10,000 identical, massive SELECT queries at the exact same moment. The database runs out of connections or exhausts its CPU and dies.
Q:To stop the stampede, you need to ensure only ONE thread fetches the data from the database, while the other 9,999 threads wait. How do you implement this at the application layer? Reveal â–¾
You implement a Mutex (Mutual Exclusion Lock) around the cache miss logic.
When a thread experiences a cache miss, it attempts to acquire a lock (e.g., using a distributed lock or an in-memory lock if tied to a single instance) for that specific cache key.
- The Winner: The thread that acquires the lock queries the database, updates the cache, and releases the lock.
- The Losers: The other 9,999 threads fail to get the lock. Instead of hitting the database, they enter a brief
sleep()loop, waking up periodically to check the cache again until the winning thread populates it.
Q:The lock works perfectly for the celebrity profile. However, on January 1st, a massive overnight batch job updates the underlying data for 500,000 users and blindly clears all 500,000 keys from the cache simultaneously. The database crashes again. Why didn’t the mutex save you? Reveal â–¾
Because you transitioned from a Cache Stampede to a Cache Avalanche.
The mutex prevents multiple threads from querying the same key. But in an avalanche, hundreds of thousands of different keys are missing simultaneously. The application will acquire 500,000 distinct locks and execute 500,000 distinct database queries in parallel. The database is still overwhelmed, just by a diverse set of queries rather than identical ones.
Q:How do you architect the caching strategy to mathematically prevent a Cache Avalanche from ever happening, even if a batch job updates all the data at once? Reveal â–¾
You must implement TTL Jitter and Background Refreshing.
First, never assign a uniform expiration time to a massive block of keys. If the base TTL is 1 hour, add a randomized jitter (e.g., $\pm 10$ minutes) so the keys expire smoothly over a 20-minute window rather than at a single exact second.
Second, for highly critical data, you abandon TTL-based expiration entirely on the read-path. Instead, you serve stale data from the cache indefinitely, and use an asynchronous background worker (or a database trigger) to proactively push updated values into the cache before they are requested.
Variations & Real-World Impact
- Operating Systems (The Original Herd): The term “Thundering Herd” originated in OS kernel design. If 100 threads are blocking on
accept()waiting for a network socket connection, and a single client connects, the OS wakes up all 100 threads simultaneously. Only one thread gets the connection; the other 99 wake up, fail, and go right back to sleep, wasting massive amounts of CPU context-switching time. Modern kernels fix this by only waking a single thread (e.g., usingepollwithEPOLLEXCLUSIVE). - Probabilistic Early Expiration (XFetch): An advanced algorithm to prevent stampedes involves reading the cache and, as the TTL approaches zero, probabilistically deciding to recompute the value early. The closer the key is to expiration, the higher the mathematical probability that a random user’s thread will take the hit to refresh it in the background before it actually expires for everyone else.
Discussion & Comments