The Outbox Relay
The database and the event stream can never disagree — because they commit together.
The incident
The order saved. The 'order created' event publish failed — a two-second broker hiccup. No error reached the user; the order existed. But fulfillment, invoicing, and analytics all listen to events, and to them this order never happened. It sat in the database, real and invisible, until a customer asked where their delivery was.
Why the obvious approach fails
Writing to the database and publishing to a broker are two operations against two systems, and no amount of careful ordering makes them atomic. Publish after commit, and a publish failure orphans the event. Publish before commit, and a rollback broadcasts something that never happened. Retry loops and try/catch narrow the window without closing it — and a window that's open one time in ten thousand is open daily at scale. The transactional outbox closes it structurally: the event is written to an outbox table *in the same database transaction* as the state change. One commit, both facts. A relay then publishes from the outbox — and if the relay dies, events wait; they don't vanish.
The standard
- State changes and their events commit in one database transaction — the event row in the outbox table is part of the write, not a follow-up.
- A relay process publishes from the outbox to the broker, marking rows published only on broker acknowledgment.
- Delivery is at-least-once by design; consumers deduplicate by event ID (pairing with the Idempotency Key discipline).
- Outbox lag — commit-to-publish time — is a monitored SLO; a stalled relay alerts before downstream systems notice silence.
- Events are immutable and ordered per aggregate, so consumers can rebuild state from the stream when they need to.
How Cortex validates it
At plan time
event-driven blueprints route every domain event through the outbox; direct publish-after-commit designs fail plan review.
At validation
assert: state_change && event_row → same_transactionassert: broker_down(window) → events_queued, zero_lostassert: outbox_lag < declared_sloAssertion syntax is illustrative.
At review
the approver sees the chaos run: broker killed mid-flow, writes continuing, the backlog draining completely on recovery — with a zero-loss count on the record.