Validation contracts: why 'the tests passed' isn't 'it's correct'

“The tests passed” is one of the most reassuring sentences in software, and one of the most misleading. Tests check what someone thought to check. They say nothing about the behavior no one wrote a test for — including the behavior you actually wanted. When a human writes the code, that gap is uncomfortable but survivable, because the author carries a private model of what “done” means. When an AI writes the code, that private model isn’t there. You have to make it explicit, or you’re trusting output against a standard that was never stated.

A validation contract is that explicit standard.

What a validation contract is

A validation contract is a definition of done for a specific piece of work — written down, before the work is accepted as finished. It captures three things:

• Behaviors — what the change must actually do, in terms of observable outcomes.
• Quality bars — performance, security, accessibility, and reliability thresholds it has to meet.
• Constraints — what it must not do, what it must not touch, and the boundaries it has to respect.

The point isn’t more tests. It’s a clear, checkable statement of what “correct” means for this change, so that “the system says it’s done” is a claim you can actually verify instead of a hope.

Why this matters more with AI

A capable engineer fills gaps in a ticket with judgment and context. An AI agent fills them with whatever’s statistically plausible. That’s fine when the spec is complete and disastrous when it isn’t. The way you close that gap is not by writing longer prompts — it’s by defining done precisely enough that the result can be measured against it, automatically, every time.

A validation contract turns “trust me” into “check this.” It moves the question from did the model behave? to does the output satisfy the contract? — and the second question has an answer.

”The tests passed” in its proper place

Tests are part of a validation contract, not a substitute for it. Passing tests are necessary evidence; they are not sufficient proof. A contract puts tests alongside the other things that decide whether work is genuinely finished — and makes the whole bar visible, so reviewers, agents, and stakeholders are all measuring against the same definition.

This is foundational to how Cortex decides that a piece of work is done. Work isn’t finished because it ran. It’s finished because it satisfied the contract that said what finished meant. Define done well, and AI-built software becomes something you can actually trust. Leave done implicit, and you’re shipping on faith.