The Circuit Breaker
One vendor's bad day stays one vendor's bad day.
The incident
The SMS provider didn't go down — worse, it got slow. Every request thread that called it waited the full 30-second timeout. Threads pooled up waiting, the pool exhausted, and requests that never touched SMS started failing. A degraded notification vendor took down checkout.
Why the obvious approach fails
Timeouts alone don't protect you — they define how long each thread is held hostage. Under sustained slowness, hostage-taking outpaces release, and resource exhaustion propagates the failure to unrelated paths. That's the cascade. The insight of the breaker pattern is that after enough failures, the kindest thing to do is stop asking: fail instantly with a fallback, give the dependency room to recover, and probe carefully before trusting it again. It's a state machine, and the states have to be enforced, not intended.
The standard
- Every external dependency call is wrapped in a breaker; naked SDK calls from business logic are banned.
- Error-rate and latency thresholds trip the breaker to open: calls fail fast with a declared fallback, no timeout hostage-taking.
- Half-open state probes with limited test traffic before closing — recovery is verified, not assumed.
- Fallbacks are designed per dependency (queue for later, degrade the feature, alternate provider) — 'throw exception' is not a fallback.
- Breaker state changes are observable events, on dashboards and in alerts.
How Cortex validates it
At plan time
The blueprint enumerates every external dependency with its thresholds and its designed fallback.
At validation
assert: external_calls.wrapped == 100%assert: dependency_slow(30s) → unrelated_paths.latency_unchangedassert: open_state.calls == fail_fast(fallback)Assertion syntax is illustrative.
At review
The approver sees the latency-injection run (with the Chaos Monkey): a dependency degraded on purpose, the breaker tripping, the platform unaffected.