Risk-Based Testing for AI: Where to Spend Your QA Budget

Risk-based testing is taught as likelihood times impact. For AI features that product is missing its most important term — because the characteristic LLM failure is not loud, and a failure nobody notices costs far more than the same failure caught on sight.

Penhale Mutual's QA lead had two hundred test prompts for the claims-triage assistant and a suspicion the set wasn't working. The prompts weren't bad. They'd accumulated the way these sets do: somebody thought of an input, wrote it down, moved on.

Sorting them by which failure each one would actually detect took an afternoon and produced an uncomfortable histogram. 143 of the 200 detected exactly one thing: a summary that reads poorly. That failure is real, and it is also the failure an adjuster spots in four seconds and fixes by rewriting a sentence.

Zero prompts detected the assistant recommending a fast-track handling path on a claim that should have gone to investigation. That failure looks completely normal on the screen. It is discovered, if it is discovered, in a reserve review a quarter later.

The unit is one named failure mode, not one test case

A test case is a thing you wrote. A failure mode is a way the feature hurts someone. Test sets grow by test case and are therefore weighted by what was easy to imagine, which is dominated by ordinary traffic doing ordinary things.

So the ranking happens at the failure-mode level, and it's a different question from where a bug lives, from whether your scores are trustworthy, and from what refuses a release. Those all assume you already know what you're protecting against. This is where you decide.

The test you can run: for each of your existing test cases, write down the one failure it would detect. Then count by failure. The shape of that histogram is your real test strategy, whatever the strategy document says.

Detectability is the term the formula is missing

Likelihood times impact comes from a world where failures announce themselves. A crashed service pages someone. A 500 shows up on a graph. Deterministic software mostly fails loudly, so how-would-we-find-out was a background assumption rather than a variable.

An LLM feature breaks that assumption, because its most common failure is fluent. A wrong policy code, an invented exclusion, a plausible summary of a different claim — none of these look like errors. They look like output. There is no exception, no status code, and nothing on a dashboard, which means the time-to-discovery is not minutes but whatever your slowest audit cycle happens to be.

That difference belongs in the score, not in a footnote:

Risk = likelihood × impact × time-to-discovery

Where the third term is measured in the units it actually takes: seconds if the user sees it immediately, a shift if a colleague catches it downstream, a quarter if it surfaces in an audit, never if nothing in the system would ever contradict it.

Here is Penhale's register after the rescore. The columns that changed the ranking are the last two.

Failure mode	Likelihood	Impact	Who'd notice, how fast	Priority
Wrong handling path on a large claim	Low	Severe	Nobody. Reserve review, ~1 quarter	P1
Fabricated policy exclusion	Med	Severe	Adjuster, only if they re-read the policy	P1
Pulls another customer's file	Low	Severe	Sometimes never — names look plausible	P1
Prompt injection via claim narrative	Low	Severe	Immediately, if it's obvious. Not if it isn't	P1
Cost per claim creeps up	High	Moderate	Finance, next invoice	P2
Summary reads poorly	High	Minor	Adjuster, ~4 seconds	P3

The bottom row had 143 tests. The top row had none. Neither of those was a decision anybody made; they were the residue of what came to mind on the days people wrote prompts.

The test: for each failure mode, name the specific person or system that would notice, and how long it would take. If the honest answer is "nobody," that mode is P1 regardless of how unlikely it is — because the cost of an undetected failure isn't the incident, it's the whole population of claims processed the same way before anyone looked.

Testing a quiet failure means building the contradiction

Once detectability is in the score, the work changes shape. A loud failure needs a test that triggers it. A quiet failure needs something in the system capable of disagreeing with the model — because if nothing can contradict the output, no amount of testing produces a signal.

Penhale's fast-track failure got three things, and only the first is a test in the usual sense:

• A targeted eval slice. Twenty-two cases built specifically from claims that should escalate — high value, prior claims history, ambiguous cause. Small, hard, and reported as its own number rather than blended into an average that the easy 240 would dominate.

• A cheap deterministic cross-check. Claim value over the authority limit plus a fast-track recommendation is now an assertion, not a judgment. It doesn't need a model and it fires in production, not just in CI.

• A sampling loop with a human. Ten fast-track recommendations a week, re-reviewed by a senior adjuster who records agreement. That's not a test suite; it's a manufactured contradiction, and it converts an undetectable failure into a measurable rate.

The third one is the piece teams skip, and it's the one that changes time-to-discovery from a quarter to a week. You cannot test your way out of a failure mode that nothing in the system can perceive — you have to add the perceiver first.

The test: pick your quietest failure mode and ask what fact, held anywhere in your system, would contradict a wrong answer. If nothing would, your first task isn't a test, it's a cross-check.

Write down what you are not testing

The output of this exercise is not only a list of things to test hard. It's a list of things you've decided to under-test, on the record, with a reason.

"We accept P3 style variance with monitoring only; revisit if adjuster escalations exceed X per week" is a sentence a team can defend in a postmortem. A silent gap is not — and the difference between the two is entirely whether somebody wrote it down before the incident rather than after.

This is also what makes the register survive contact with pressure. When a release is late and testing is the thing being compressed, an unwritten priority order collapses into whatever the loudest person remembers. A written one gets argued with, which is the point.

The test: find the lowest-priority mode on your register and check whether the decision to under-invest is written anywhere someone outside QA could read it. If not, it isn't a decision yet.

When ranking is the wrong move

• Fewer than about five failure modes. Just test all of them. A register is overhead you're paying to avoid work smaller than the overhead.

• Before you know the failure modes. Ranking a list you invented ranks your imagination. Break the feature by hand for a day first; the register is the second artifact, after the cost-of-wrong you named in discovery.

• Regulated or safety-critical scope. Where an external standard names required checks, those are floor, not candidates. Rank what's left over.

• When impact is genuinely unknown. A brand-new product surface has no evidence behind its impact scores, so the ranking is a guess wearing a table's authority. Ship narrow, watch, then rank.

The test: ask whether your impact scores came from evidence or from a meeting. Meeting-derived scores are a reasonable starting point and a bad thing to still be using a year later.

What to change this week

Don't rewrite the test set. Sort it.

Take your existing cases and label each with the single failure it would detect, then count. The histogram takes an afternoon and it is usually the most persuasive artifact you will produce this quarter — nobody argues with their own test set's shape.

Then add the third column to your register. For every failure mode, write who would notice and how fast. The rows where the answer is "nobody" are your real priorities, and they will not be the rows you were testing.

Penhale's set is 90 prompts now, down from 200. The fast-track slice has 22 cases, the authority-limit cross-check runs on every recommendation in production, and a senior adjuster reviews ten a week. The style prompts that were 143 of 200 are 12, which turned out to be plenty — because the failure they detect was always going to be caught by a human in four seconds.