Definition of Done for an AI Feature
- Shawn West
- 4 days ago
- 5 min read
The feature shipped on a Thursday because the demo was clean. Someone typed three reasonable questions, the model answered all three well, the room agreed it was ready, and it went out. By Monday, support had a queue of screenshots: the model confidently citing a refund policy the company had never had. Nobody had lied and nobody had been careless. The team had just used the only Definition of Done they had — "it worked when we tried it" — and for an AI feature, that definition is worthless, because the whole risk lives in the inputs you didn't try.
A Definition of Done is a team-wide checklist: the baseline every piece of work must clear before it's shippable, independent of any one feature. Most teams have one for normal features — tested, reviewed, deployed, documented. Almost none have one for AI features, so AI features get shipped on the demo. This guide is that missing checklist: what "done" has to mean when the output is a judgment a model made and not a branch you wrote.
Why the normal Definition of Done isn't enough
Your existing DoD assumes deterministic behavior. "Tests pass" means something when the same input yields the same output. For an AI feature, the same input can yield a different output tomorrow — after a model update you don't control, a prompt tweak, or just temperature. "It passed review" means something when a human can read the code and see the behavior. For an AI feature, the behavior isn't in the code; it's in a distribution of outputs you can only sample.
So the AI DoD isn't a replacement for your normal one — it's an extension. The normal items still apply (it's reviewed, it's deployed, it's instrumented). These are the additional gates that a non-deterministic, judgment-producing feature has to clear before "done" is a fact instead of a hope.
The Definition of Done for an AI feature
## Definition of Done — AI Feature
Scope & target
[ ] The feature has a written scope: the decision it feeds and a "good output" definition
[ ] There is an eval set of real inputs with known-good expectations (incl. hard cases)
Quality evidence (not vibes)
[ ] Evals run in CI and the feature meets an agreed pass threshold — not a one-off demo
[ ] Known failure modes (plausible-but-wrong) each have a test that would catch a regression
[ ] A human reviewed a sample of outputs on inputs the builder did NOT choose
Failure handling
[ ] Defined behavior when the model is wrong, uncertain, or unavailable (fallback / "I don't know")
[ ] Guardrails for the unacceptable failures (no invented policy/number/citation ships silently)
[ ] The human-in-the-loop role from scope is actually implemented (approval step exists if promised)
Operations
[ ] Cost per call and expected volume are known and within budget
[ ] Latency is measured and acceptable for the workflow
[ ] Outputs are logged so failures can be captured and replayed
[ ] There is a rollback / kill switch that turns the feature off without a deploy
Ship decision
[ ] Cost-of-wrong from scope matches the review level shipped (high cost => human approval)
[ ] Someone who is NOT the builder signed off on the evidence, not the demo
You don't need every box for every feature — a low-stakes internal helper and a customer-facing autonomous agent are different risk profiles. But the checklist forces the conversation: if you're skipping the rollback switch, that's now a decision on the record, not an oversight discovered during the incident.
The four gates that actually catch the Monday-morning failure
Of the list above, four items are the ones that would have stopped the refund-policy incident. If you adopt nothing else, adopt these.
1. An eval set, run in CI — not a demo. The demo tested three inputs the builder chose. An eval set tests the inputs you're afraid of, every build, and fails the pipeline when quality drops. This is the difference between "it worked when I tried it" and "it works on the cases that matter, and I'll know the moment it stops." A demo proves a feature can succeed; an eval proves it usually does.
2. A test for each plausible-but-wrong failure. "Invents a policy that doesn't exist" is a specific, nameable failure. It should have a test — an input that tempts the model to hallucinate a policy, asserting it doesn't. Vague QA ("we spot-checked it") does not survive a model update. A named test does.
3. Defined behavior for wrong / uncertain / unavailable. The refund feature had no answer for "the model doesn't actually know" — so it guessed, fluently. Done means you decided what happens when the model is unsure (say "I don't know," route to a human, return the source) and what happens when the API is down. An AI feature without a defined failure behavior isn't done; it's a confident guess with good uptime.
4. A rollback that doesn't need a deploy. When an AI feature goes wrong it often goes wrong broadly and suddenly — a model change shifts every output at once. You need to turn it off in seconds, via a flag, without shipping code. If the only way to stop a bad model is a deploy, your incident is as long as your pipeline.
"This is too heavy for our little feature"
Then scale it, don't skip it. The checklist is a menu keyed to the cost-of-wrong you already wrote down in scope. An internal tool that drafts text for a human to edit needs the eval set and the failure behavior; it may not need a formal sign-off gate. A feature that acts on customers' money or health needs all of it and then some. The mistake isn't having a heavy DoD for a light feature — it's having no DoD and letting the demo decide, which is exactly how the light-looking feature ships the policy it invented.
The point of a Definition of Done is that the ship decision is made against evidence you agreed to in advance, when it was cheap to be honest — not against a clean demo in a room full of people who want to ship.
Key Takeaway
"It demos well" is the Definition of Done that ships hallucinated policies on Monday, because the demo only ever tests the inputs the builder chose. Extend your team's normal DoD for AI features: a written scope and an eval set, evals in CI against a threshold, a regression test per named failure mode, defined behavior for wrong/uncertain/unavailable, cost and latency known, output logging, a no-deploy rollback, and a non-builder signing off on the evidence instead of the demo. Scale the checklist to the cost of being wrong — but make skipping any item a decision on the record, not a gap you find during the incident. The refund feature didn't ship broken because the team was sloppy. It shipped broken because "it worked when we tried it" was the only bar it had to clear.
