The Floodgate
Ingestion decoupled from processing, so a traffic spike is a queue depth — not an outage.
The incident
An exam platform. 100,000 students, one submission deadline, one relational database. At 11:59 the writes arrived faster than rows could lock. Connections exhausted, the load balancer timed out, and the platform died at the only moment of the day that mattered.
Why the obvious approach fails
The naive architecture couples accepting a request with processing it: the web server holds the connection open while the database does the work. That coupling is invisible at 100 requests per second and fatal at 50,000, because a database under lock contention doesn't slow down linearly — it collapses. Threads pile up waiting, the connection pool drains, and health checks start failing, which takes out even the traffic the system could have served. Adding servers doesn't help; they all queue on the same database. The bottleneck isn't capacity. It's the synchronous shape of the flow.
The standard
- Spike-prone write paths never touch the primary database synchronously; they land in a durable stream (Redis Streams / Kafka) with sub-5ms acknowledgment.
- A decoupled consumer fleet drains the queue at a measured rate the database is proven to sustain.
- Back-pressure is explicit and user-visible: when the queue passes threshold, clients get position feedback, not connection errors.
- Queue depth, drain rate, and consumer lag are first-class metrics with alerts — not afterthoughts.
- Every queued event is durable: an accepted request survives worker crashes and restarts.
How Cortex validates it
At plan time
The blueprint for any flow with spike-shaped load (submissions, checkouts, registrations) must declare its ingestion path; a synchronous write on a declared spike path fails the plan review.
At validation
assert: spike_paths.write_mode == "queued"assert: ack_latency_p99 < 5ms under load profileassert: consumer_drain_rate <= db_sustained_write_ceilingAssertion syntax is illustrative.
At review
The approver sees the load profile the flow was tested against and the queue-depth behavior at peak, as evidence attached to the Mission.