The Log Aggregator
One failed request, one search, the whole story.
The incident
A payment failed for one important customer. The request had touched five services. Five engineers opened five log files with five timestamp formats and no shared identifier. The debugging session was measured in days; the fix, once found, took nine minutes.
Why the obvious approach fails
Per-service logging answers 'what did this service see?' — but production incidents are questions about journeys: what happened to this request, across everything it touched? Without a propagated identifier, correlation is manual archaeology through clock skew and format drift; cost grows with architecture size exactly when architecture size is why you can't find anything. Tracing isn't a logging upgrade — it's a different data model: every request gets an identity at the edge, and everything downstream inherits it.
The standard
- Every request receives a correlation ID at ingress; every service, worker, and query it touches propagates it (OpenTelemetry standard).
- Logs are structured (JSON), with shared field conventions — grep-ability is a design requirement.
- All services ship to one queryable store: one search on one ID returns the full request lifecycle.
- Traces capture timing per hop, so 'where did the time go' is a lookup, not a debate.
- Sampling is tiered: errors and slow requests at 100%, healthy traffic sampled — visibility without cost explosion.
How Cortex validates it
At plan time
The blueprint declares trace propagation across every service boundary, including async hops through queues.
At validation
assert: correlation_id.propagation == 100% (incl. async)assert: single_query(id) → full_request_lifecycleassert: logs.format == structuredAssertion syntax is illustrative.
At review
The approver sees a forced mid-chain failure traced end-to-end with one search, timing per hop included.