The Immutable Forge
Environments are built from code, destroyed without fear, and identical every time.
The incident
Staging passed everything. Production failed immediately. The diff took three days to find: a kernel parameter someone had hand-tuned on the production hosts during an incident in 2023, undocumented, unrepeated anywhere else. It wasn't the first ghost setting — it was just the first one that bit. Nobody could say how many others were waiting, because nobody could say what production actually was.
Why the obvious approach fails
Hand-managed infrastructure accumulates undocumented state the way old houses accumulate wiring — every quick fix during every incident becomes a permanent, invisible divergence. 'Works in staging' stops meaning anything, because staging and production are only nominally the same system. Configuration management that patches servers in place slows the drift but can't eliminate it; the server's history still matters. Immutability eliminates history as a variable: environments are declared in code (Terraform/Pulumi-class), built from those declarations, and *replaced* rather than modified. If it's not in the code, it doesn't exist — and the incident-time hand-tweak either becomes code or disappears at the next rebuild.
The standard
- All infrastructure is declared in code, versioned, and reviewed like application code — console changes in production are break-glass events, audited and reconciled into code within a declared window.
- Environments rebuild from scratch on a schedule or per release; a rebuild is routine, not a ceremony.
- Drift detection runs continuously — the declared state and the actual state are compared, and divergence is an alert.
- Staging and production derive from the same modules with parameterized differences only; their divergence is enumerable, on purpose.
- Infrastructure changes flow through the same plan-review-apply gate as code — a terraform plan is a reviewable diff.
How Cortex validates it
At plan time
infrastructure changes appear in Mission plans as declared-state diffs, reviewed before apply.
At validation
assert: drift(declared, actual) == 0 (or alerted)assert: environment.rebuild_from_code → functionalassert: console_changes → reconciled_within(window)Assertion syntax is illustrative.
At review
the approver sees the phoenix drill: an environment destroyed and rebuilt purely from code, coming up green — proof that the code is the whole truth.