Acceptance Criteria for Non-Deterministic Features
- Shawn West
- 3 days ago
- 6 min read
The ticket looked normal. "As a user, I want the assistant to answer questions about my invoices." The acceptance criteria underneath were the usual shape: Given a user asks about an invoice, When they submit, Then the assistant returns the correct answer. Everyone approved it. Then QA tried to sign it off and hit a wall: the same question returned a slightly different answer every time they ran it. Sometimes great, sometimes it rounded the total wrong, once it made up a due date. Was the story done? Nobody could say — because the criterion asked for "the correct answer," singular, from a feature that produces a distribution of answers, and there is no single run you can point at to prove or disprove it.
That's the trap. Acceptance criteria were invented for deterministic features, where the same input yields the same output and "Then X happens" is a fact you can check once. AI features break that assumption. Run the same prompt twice and you may get two different outputs — both valid, or one subtly wrong. This guide is how to write acceptance criteria that survive non-determinism: criteria a tester can actually verify, and a story you can actually call done. It's the story-level companion to scoping the feature (Discovery for AI Features) and the team-wide bar for shipping it (Definition of Done for an AI Feature).
Why "Then the output is correct" fails
Three things break when you paste a normal criterion onto an AI story:
"The output" implies one output. There isn't one. There's a range. A criterion that names a single expected result is unverifiable the moment the second run disagrees with the first.
"Correct" implies a single truth. For "summarize this," "extract the total," or "answer this question," there are many acceptable outputs and many unacceptable ones, and the line between them is a judgment, not a string match. "Correct" doesn't tell the tester where that line is.
One run proves nothing. With a deterministic feature, one passing run is the proof. With a probabilistic one, a single good run is a coin landing heads — it doesn't tell you the coin is fair. You have to specify behavior over a set of runs, not a single execution.
So the fix isn't better wording on a single-run criterion. It's changing what the criterion asserts.
The three shapes of a verifiable AI criterion
Most AI acceptance criteria fall into one of three shapes. Pick the shape that matches the behavior; don't force everything into Given-When-Then.
1. Invariant criteria — "must always / must never"
Some things must hold on every output, no exceptions. These are your hard guardrails, and they're the most valuable because a single violation is a real failure, testable on one run.
- The response NEVER invents an invoice number, amount, or date not present in the source data
- The response ALWAYS cites the invoice ID it used
- When the answer isn't in the data, the assistant says so — it never guesses
- No output contains another customer's data
Invariants are pass/fail and unambiguous. Write these first — they're where the real risk lives.
2. Distributional criteria — "on a set, at least X%"
For quality that's a matter of degree, assert behavior over an eval set, not a single input. This is the criterion shape that non-determinism forces on you, and it's the one teams most often miss.
- On the 40-question eval set, ≥ 90% of answers are rated correct by the rubric
- On the 15 "hard" invoices (partial data, foreign currency, credits), ≥ 80% correct
- Zero answers in the set violate an invariant above (a single violation fails the story)
Note what this requires: an eval set with known-good expectations, which you should already have from discovery. The criterion isn't "it's correct" — it's "it clears an agreed bar on inputs we chose in advance, including the ones we're afraid of."
3. Boundary criteria — "when uncertain / when it fails"
Specify the unhappy paths explicitly, because a non-deterministic system will hit them and its default is to respond confidently anyway.
Given a question the invoice data cannot answer
When the assistant responds
Then it states it doesn't have that information and offers to hand off — it does not fabricate
Given the model API is unavailable
When a user asks a question
Then the user sees a graceful fallback, not a spinner or a stack trace
Given-When-Then still works here — because you're specifying a category of situation and a required category of response, not pinning one exact string.
A worked example
The invoice story, rewritten so it can actually be signed off:
## Story: Invoice Q&A assistant
As a user, I want to ask questions about my invoices and get answers grounded in my data.
## Acceptance Criteria
Invariants (every output):
- Never states an amount, date, or invoice number not present in the source data
- Always references the specific invoice(s) used
- When the data doesn't contain the answer, says so explicitly — never guesses
Distributional (on the 40-question eval set):
- ≥ 90% of answers rated correct by the review rubric
- 100% of answers comply with the invariants above
- ≥ 80% correct on the 15-item "hard" subset
Boundary:
- Unanswerable question -> states the limitation, offers human handoff
- Model/API unavailable -> graceful fallback message, no raw error
- Ambiguous question -> asks a clarifying question rather than guessing which invoice
Now "done" is a decision anyone can check: run the eval set, confirm the percentages and the zero-violation invariants, exercise the three boundary cases. No single-run coin flip. No argument in the sign-off meeting about whether one particular answer counted.
Where the rubric comes from
Distributional criteria lean on a phrase — "rated correct by the rubric" — that's doing real work, so make it real. The rubric is the written definition of what "correct" means for this feature: the hand-authored good/bad examples from discovery, turned into a checklist a human (or an LLM-as-judge) applies the same way every time. "The reviewer felt it was good" is not a rubric; "the answer states the correct total, cites the invoice, and adds no unsupported claims" is. Without a rubric, distributional criteria are just a percentage with no defined numerator — and you're back to arguing about vibes, only now with a spreadsheet.
Anti-patterns
Single-output criteria. "Then it returns the summary." There is no the. Assert properties of the summary, or a pass rate over a set.
Unmeasurable quality words. "Helpful," "natural," "high-quality" — not verifiable. Convert to a rubric item or delete.
Ignoring the tails. Criteria that only cover the happy path. For AI, the tail (uncertain, adversarial, out-of-scope input) is where it fails and stays confident. Boundary criteria are not optional.
100% thresholds on judgment tasks. Demanding 100% correctness on an open-ended task either blocks the story forever or trains everyone to fudge the rubric. Set an honest bar; enforce invariants at 100%, quality at a realistic percentage.
A rubric that lives in someone's head. If the definition of "correct" isn't written down, your pass rate isn't reproducible — a different reviewer, or the same one on a tired afternoon, will score it differently.
Key Takeaway
Acceptance criteria assume one input yields one checkable output — and AI features violate that assumption, which is why "Then the output is correct" can't be signed off. Write three shapes instead: invariants that must hold on every output (never fabricate, always cite, say "I don't know" rather than guess), distributional criteria that assert a pass rate on a pre-agreed eval set including the hard cases, and boundary criteria for uncertain, failing, and out-of-scope inputs. Back the distributional bar with a written rubric so "correct" means the same thing every time. The invoice story wasn't un-sign-off-able because the feature was bad — it was because "the correct answer" asked a probabilistic system for a deterministic promise. Change what the criterion asserts, and "done" becomes something you can prove instead of something you argue about.
