Risk-Based Testing for AI: Where to Spend Your QA Budget

The QA lead had a spreadsheet of two hundred test prompts and a growing unease. They'd been added the way most AI test sets grow — someone thought of an input, wrote it down, moved on. The prompts weren't bad. They just weren't chosen. And the feature kept getting surprised in production by failures that felt, in hindsight, obvious: the one input type that touched money, the one phrasing that made the model leak its instructions. Meanwhile a third of the two hundred prompts were minor variations on "summarize this normal document," testing the same low-risk behavior over and over. The set was large and busy and pointed in the wrong direction, because nobody had asked the only question that scales when the input space is infinite: of everything that could go wrong, what would hurt the most and happen the most?

That's risk-based testing, and AI needs it more than deterministic software does — because you can enumerate the branches of a function, but you cannot enumerate the inputs to a model. The input space is effectively infinite and your QA time is finite, so coverage-by-volume is a fantasy. The only sane strategy is to prioritize: rank failure modes by risk and spend your testing where risk is highest. This is how to do that for an AI feature — identify the failure modes, score them, and allocate testing to match. It operationalizes the cost-of-wrong you named in Discovery for AI Features.

Risk = likelihood × impact

The whole method rests on one definition. The risk of a failure mode is how likely it is to happen multiplied by how much it hurts when it does. Test in priority order of that product — not in the order failures occurred to you.

• High impact, high likelihood → test heavily, gate on it, monitor in production.

• High impact, low likelihood → still test; low odds don't save you from a catastrophic single occurrence (a data leak that happens once is not "low risk").

• Low impact, high likelihood → test enough to keep it in check; don't over-invest.

• Low impact, low likelihood → acknowledge and mostly ignore. This is where the spreadsheet's hundred summarize-variations were living.

The spreadsheet failed because it was sorted by "what came to mind," and what comes to mind is dominated by the normal, low-impact happy path. Risk sorts by consequence.

Step 1 — Enumerate AI failure modes

You can't rank what you haven't named. AI features share a recognizable set of failure modes — start here and add feature-specifics:

- Fabrication      — invents facts, policies, citations, numbers
- Wrong retrieval  — reasons over the wrong / stale / someone else's data
- Prompt injection — user input overrides system instructions
- Data leak        — exposes other users' data, secrets, or the system prompt
- Silent error     — fails without signaling; looks confident and complete
- Overreach        — takes an action it wasn't authorized to take (agents)
- Cost/latency blowup — a query pattern that explodes spend or response time
- Bias / harm      — outputs that are unfair or unsafe for a group of users
- Graceful-fail gap — no defined behavior when the model is unsure or down

Not all apply to every feature. A read-only summarizer can't overreach; an agent with tools very much can.

Step 2 — Score each mode for this feature

Rate likelihood and impact (a simple High/Med/Low is enough to start). Impact is anchored by the cost-of-wrong from discovery — who absorbs the failure and how badly.

## Risk register: <feature = customer-facing invoice assistant>

Failure mode        | Likelihood | Impact | Priority
--------------------|-----------|--------|---------
Data leak (cross-tenant) | Low  | High   | P1  (catastrophic once)
Fabricated amount/date   | Med  | High   | P1
Wrong retrieval          | Med  | High   | P1
Prompt injection         | Med  | Med    | P2
Silent error             | Med  | Med    | P2
Cost/latency blowup      | Low  | Med    | P3
Bias                     | Low  | Low    | P3

The register turns "we should test more" into a ranked, defensible list — and makes the trade-offs visible to everyone, not buried in a QA lead's instinct.

Step 3 — Allocate testing to the ranking

Now spend the budget where the register points:

• P1 modes → dedicated invariant tests (zero-tolerance), a slice of the eval set built specifically to trigger them, a release-gate block, and production monitoring. The cross-tenant leak gets its own targeted suite, not one prompt among two hundred.

• P2 modes → covered in the eval set and red-team pass; monitored, but not necessarily gating.

• P3 modes → a lightweight check and an acknowledgment in the register. Deliberately under-invested — on purpose, on the record.

Notice the inversion from the spreadsheet: the invoice assistant now has more tests pointed at data leak and fabricated amounts than at generic summarization, because that's where likelihood-times-impact is highest. The set gets smaller and far more protective.

Step 4 — Make the under-investment explicit

The point of risk-based testing isn't only to test the top risks hard. It's to decide, on the record, what you're not testing much — so a low-priority gap is a documented choice, not a silent hole discovered in an incident. "We accept P3 bias risk with monitoring only, revisit if usage changes" is a sentence a team can stand behind. An untested gap nobody named is the one that becomes the postmortem.

Keeping the register alive

Risk isn't static. A model update can raise a likelihood; a new user segment can raise an impact; a production incident is direct evidence a score was wrong. Revisit the register on model changes, on scope changes, and after every incident — and feed real failures back in. A risk register you wrote once and never reopened is a snapshot of what you feared at launch, not what threatens you now.

Key Takeaway

An AI feature's input space is infinite and your QA time isn't, so testing by volume is a fantasy — a big test set sorted by "what came to mind" points mostly at the low-risk happy path while the money-touching and instruction-leaking inputs go under-tested. Risk-based testing fixes the aim: enumerate the failure modes (fabrication, wrong retrieval, injection, data leak, silent error, overreach, cost blowup, bias, graceful-fail gaps), score each by likelihood × impact for this feature, and allocate testing to the ranking — P1 modes get invariants, targeted evals, a gate block, and monitoring; P3 modes get a documented shrug. Then make the under-investment explicit, and keep the register alive as models and usage change. The QA lead's two hundred prompts weren't too few. They were pointed at the wrong things — because the set was never sorted by what a failure would actually cost.

---

Part of the Quality Engineering for AI path. See also: Discovery for AI Features and A Test Strategy for LLM Features.