The Connection Marshal
The database sees a calm, fixed number of connections — no matter how wild the app tier gets.
The incident
Traffic spiked, autoscaling did its job, and twelve new app instances came online — each opening its own fifty connections. Postgres hit max_connections and started refusing everyone, including the existing healthy instances, including the on-call engineer's psql session trying to diagnose it. The autoscaler, seeing errors, scaled further. The fix made the outage.
Why the obvious approach fails
Postgres connections are expensive — each is a process with real memory — so the database's connection ceiling is low and hard. Meanwhile the modern app tier is elastic by design: instances multiply exactly when load arrives. Left uncoordinated, these two facts collide at the worst time — scaling the app tier scales the connection count into the ceiling, and the database punishes everyone equally. The marshal breaks the coupling: applications connect to a pooler (PgBouncer-class), which multiplexes thousands of client connections onto a small, fixed set of database connections. The app tier scales freely; the database's world stays constant.
The standard
- All application traffic reaches the database through a pooler; direct connections are reserved for administration and are budgeted.
- Pool sizes are derived from database capacity, not from app instance counts — total connections are constant under autoscaling.
- Pooling mode (transaction vs session) is a deliberate choice per workload, with the feature constraints (prepared statements, advisory locks) documented.
- Connection acquisition time is a monitored metric — pool exhaustion is an alert with a runbook, not a mystery.
- Per-service pool budgets prevent one service's leak from starving the rest (pairing naturally with the Tenant Governor on shared clusters).
How Cortex validates it
At plan time
blueprints declare the pooling topology and per-service budgets; direct app-to-database connection designs fail review.
At validation
assert: app_connections.via == poolerassert: autoscale(app_tier × 4) → db_connections.delta == 0assert: pool_exhaustion → alert && graceful_queueingAssertion syntax is illustrative.
At review
the approver sees the scale drill: the app tier quadrupled, the database's connection graph flat, acquisition latency within budget.