The Blob Ingestor
Big files go straight to storage — the server signs the permission slip and steps aside.
The incident
A 500MB video assignment on a 3G connection held a server thread hostage for twenty minutes. A classroom's worth of simultaneous uploads exhausted the pool, and the site went down for everyone — including people who weren't uploading anything. At minute nineteen of one upload, the connection dropped. The student started over.
Why the obvious approach fails
Proxying uploads through the application server means the server's most expensive resource — a connection-holding thread — is occupied for exactly as long as the user's network is slow. Capacity becomes a function of your worst clients. And a single-shot upload makes failure total: one dropped packet at 99% wastes everything before it. The fix is structural: the server's job is authorization, not transport. Storage providers already accept uploads directly, resumably, in parallel — the server just has to sign the permission and track the pieces.
The standard
- Files travel client-to-storage directly via presigned, time-limited URLs; application servers never proxy file bytes.
- Large files upload in ~5MB chunks, in parallel, with per-chunk verification (ETags) and server-side assembly on completion.
- Pause and resume are native: an interrupted upload resumes from the last confirmed chunk, never from zero.
- Presigned URLs are scoped tight — one key, one operation, short expiry — so a leaked URL is a small, dying thing.
- Post-upload processing (scanning, transcoding) triggers by event, decoupled from the upload itself.
How Cortex validates it
At plan time
any upload feature declares its size classes, chunking strategy, and URL scoping.
At validation
assert: file_bytes.through_app_server == 0assert: interrupted_upload → resume_from(last_chunk)assert: presigned_url.scope == [single_key, single_op, short_ttl]Assertion syntax is illustrative.
At review
the approver sees the hostile-network run: a large upload interrupted repeatedly and resuming, server thread usage flat throughout.