The Retry Helix
Retries that help the system recover instead of finishing it off.
The incident
The API came back from a two-minute outage — and immediately went back down. Every client that had failed during the window retried at the same instant, in lockstep, forever. The outage wasn't the incident. The synchronized recovery was.
Why the obvious approach fails
Immediate retry turns N failed clients into an N-sized battering ram. Plain exponential backoff is better but keeps clients synchronized: everyone who failed at T retries at T+1, T+2, T+4 — as one coordinated wave. The fix is mathematical, not moral: jitter. Randomizing each client's delay decorrelates the fleet, turning waves into a gentle drizzle the recovering server can absorb. And retries without idempotency remain dangerous regardless of schedule — a retry that re-executes a success is a bug with a timer.
The standard
- All network retry logic flows through one layer; ad-hoc retry loops in feature code are banned.
- Backoff is exponential with full jitter (randomized within the growing window), with a declared cap and retry budget.
- Every retried request carries the same Idempotency-Key as the original — retry and exactly-once are one system.
- Retries respect server signals: Retry-After is honored, and circuit-breaker state short-circuits pointless attempts.
- Retry storms are observable: retry rate per endpoint is a dashboard metric with alerting.
How Cortex validates it
At plan time
The blueprint declares the retry policy (budget, cap, jitter) per integration; hand-rolled retry loops fail plan review.
At validation
assert: retry_delays.distribution == jittered (no synchronization)assert: retried_requests.idempotency_key == originalassert: retry.respects(Retry-After, circuit_state)Assertion syntax is illustrative.
At review
The approver sees the recovery simulation: server restored after mass failure, retry arrival spread over the window, no secondary collapse.