The Chaos Monkey
Break it on purpose in staging, or production will break it for you — on Black Friday.
The incident
Every resilience mechanism was in place: breakers, retries, queues. All of them configured, none of them ever exercised. The first real network partition hit at peak traffic and found the one misconfigured timeout in minutes — something a single staged failure drill would have found for free.
Why the obvious approach fails
Resilience code is the least-executed code in any system: it runs only when things go wrong, which means it's tested only when the stakes are highest. 'We have circuit breakers' is a claim about code existing, not about behavior working — thresholds can be wrong, fallbacks can be broken, retry budgets can be misconfigured, and nothing in a sunny-day test suite will ever tell you. The only way to know failure handling works is to cause failures, deliberately, where the blast radius is contained.
The standard
- Failure injection runs as a scheduled discipline in staging: latency spikes, packet loss, dependency outages, killed workers.
- Every resilience mechanism has a drill that proves it: breakers trip under injected slowness, retries recover from injected loss, queues survive killed consumers.
- Drills assert user-facing behavior, not just internals — 'the platform stayed up' is the pass condition.
- Findings feed the standards: every drill failure becomes a fixed configuration and a regression drill.
- Pre-launch resilience runs are a release gate for spike-shaped or high-stakes systems.
How Cortex validates it
At plan time
The blueprint's resilience mechanisms each get a paired drill definition; a breaker with no drill is an unverified claim and fails plan review.
At validation
assert: resilience_mechanisms.drilled == 100%assert: drill(db_latency_2000ms) → user_error_rate < thresholdassert: drill(worker_kill) → jobs_recovered == 100%Assertion syntax is illustrative.
At review
The approver sees the drill report: every injected failure, observed behavior, and pass/fail against the declared thresholds — attached to the release evidence.