The Job Marshall
Background work with a memory — jobs that survive restarts, retries, and Mondays.
The incident
The monthly report generator ran as a cron job on one server. A deploy restarted that server mid-run. The job didn't fail — failing would have alerted someone. It simply ceased to exist, and the first person to notice was the customer asking where their report was, a week later.
Why the obvious approach fails
Cron plus a script is a scheduler, not an orchestrator — it starts work and forgets it exists. There's no record that a job was running, so a crash erases it; no retry semantics, so transient failures are permanent; no priority, so the OTP email queues behind the bulk export; and no dead-letter path, so poison jobs either vanish or retry forever. Background work needs the same discipline as request work: durable state, explicit lifecycle, bounded retries, and a place where failures go to be looked at rather than to disappear.
The standard
- All background work runs through a persistent queue (BullMQ/Sidekiq-class); bare cron scripts for stateful work are banned.
- Every job has a durable lifecycle — pending → active → completed/failed — that survives process and server restarts.
- Failures retry with exponential backoff to a declared budget, then land in a dead-letter queue with full context for inspection.
- Priority lanes are explicit: critical jobs (OTP, payment webhooks) never queue behind bulk work.
- Queue depth, processing latency, and DLQ arrivals are first-class metrics with alerts.
How Cortex validates it
At plan time
Every async workflow in the blueprint declares its queue, priority lane, retry budget, and DLQ handling.
At validation
assert: stateful_background_work.via == persistent_queueassert: worker_kill(mid_job) → job_recoveredassert: critical_lane.latency independent_of(bulk_lane.depth)Assertion syntax is illustrative.
At review
The approver sees the kill drill: workers terminated mid-run, every job recovered or dead-lettered with context, nothing silently gone.