The Cache Sentinel
A cache that never truly expires, and a database that never meets the herd.
The incident
A homepage cache key with a 60-second TTL. At second 61, ten thousand concurrent requests missed the cache simultaneously — and every one of them independently decided to rebuild it from the database. The cache existed to protect the database. Its expiry took the database down.
Why the obvious approach fails
Cache-aside logic — check cache, on miss read the database, write back — contains a race that only appears under concurrency: between the miss and the write-back, every other request also misses. This is the stampede (dog-pile) problem, and it's vicious precisely because it gets worse with popularity: your hottest key is the one that kills you. TTL tuning doesn't fix it; it just schedules the stampede. The fix has to change who is allowed to rebuild.
The standard
- On a cache miss for a protected key, exactly one request acquires a regeneration lock; all others serve stale data or wait — they never fall through to the database.
- Hot keys use probabilistic early refresh: the value is rebuilt in the background before expiry, so the key never publicly expires.
- Stale-while-revalidate is the default posture for read-heavy keys — serving slightly old data beats serving an outage.
- Cache hit ratio and regeneration frequency are monitored per key class; a falling hit ratio is an alert, not a surprise.
How Cortex validates it
At plan time
Any blueprint introducing a cache layer must classify keys (hot / warm / cold) and declare the stampede strategy per class.
At validation
assert: concurrent_miss(hot_key) → db_reads == 1assert: hot_keys.refresh == "early_probabilistic"Assertion syntax is illustrative.
At review
The approver sees the concurrent-miss test result: N simultaneous requests, one database read, as attached evidence.