The Auto-Scaler
Scaling on leading indicators — capacity that arrives before the spike, not after the crash.
The incident
The marketing push landed at noon. Cloud auto-scaling noticed at 12:02, when CPU crossed 80%. The new instances were serving traffic by 12:06. The outage ran from 12:01 to 12:06 — the scaler worked exactly as designed, and was exactly five minutes too late.
Why the obvious approach fails
Reactive scaling watches lagging indicators. CPU rises after load arrives; instances boot minutes after that. For gradual growth this is fine. For spike-shaped businesses — launches, exams, flash sales, breaking news — the spike front outruns the boot time every single time. The information needed to scale exists before CPU moves: it's in the queue depth, the connection rate of the load balancer, and, most powerfully, in the business calendar. A scaler that can't read those is a postmortem generator.
The standard
- Scaling triggers include leading indicators: queue depth growth rate (from the Floodgate) and request-rate acceleration — not CPU alone.
- Known business events pre-warm capacity on schedule: the noon exam gets its fleet at 11:55, declared in configuration.
- Scale-up is aggressive and scale-down is conservative — asymmetry is deliberate; thrash is a failure mode.
- Every scaling event is logged with its trigger, so capacity decisions are auditable after the fact.
How Cortex validates it
At plan time
Blueprints for spike-shaped workloads must declare expected load profiles and pre-warm schedules.
At validation
assert: scaling.triggers.include("queue_depth_rate")assert: scheduled_events → capacity_prewarmed(T-5min)Assertion syntax is illustrative.
At review
The approver sees the simulated spike run: time-to-capacity versus time-to-load, as attached evidence.