A Test Strategy for LLM Features (Not Just Evals)

"We have evals" is treated as a coverage statement. It is a statement about one layer of a system with five, and the other four contain the bugs that reach production.

Penhale Mutual's claims assistant spent nine days summarising the wrong file.

An adjuster opens a claim, the assistant retrieves the loss report and the policy schedule, and drafts a triage summary. A schema change had shifted the retrieval filter one field to the left, so for claims sharing a surname the assistant sometimes pulled a different customer's report. It then read that report carefully and summarised it accurately.

The eval suite ran nightly through all nine days. The average went up, because that week's prompt tuning was genuinely working — and because every eval case hands the model its context from a fixture. The retrieval step the bug lived in was never executed. There was no failing test to notice, and no eval score to fall.

The unit is the layer a bug can live in

An LLM feature is not a model with some code around it. It's a system, and each part fails in a way the others can't see. This post's question is narrow: given a defect, which layer produced it — and does any test you own execute that layer? That is a coverage question, and it's distinct from whether your eval numbers mean anything and from which failures deserve the budget.

Layer	What it executes	The bug it catches	What it structurally cannot see
1 — Unit	Parsing, prompt assembly, output handling, retries	Truncation, bad escaping, a fallback that never fires	Anything involving a real model call
2 — Integration	Retrieval, tool calls, external APIs	The wrong context reaching a working model	Whether the model then reasons well
3 — Evals	The model, on curated inputs	Reasoning, tone, faithfulness, regressions	Whether those inputs resemble production
4 — Adversarial	The model, on hostile inputs	Injection, extraction, being talked out of the rules	Ordinary failure on ordinary traffic
5 — Production	Everything, on real traffic	Drift, the input nobody imagined, cost creep	Anything, before it has already happened

The right-hand column is the one worth sitting with. Each layer has a blind spot that is structural — not a gap you close by writing more tests at that layer. Nine hundred eval cases would not have caught Penhale's bug, because the property that was broken was never exercised by an eval.

The test you can run: take your last three AI defects and assign each one a layer. If any landed outside layer 3, count how many tests you own that execute that layer. That count, not your eval score, is your coverage there.

A layer-2 bug can raise your eval score

This is the mechanism that makes retrieval bugs so durable, and it's worth being precise about.

An eval case is an input paired with an expectation you wrote down in advance. To keep it stable and cheap, the input is fixed — the fixture supplies the retrieved context directly, so the model is scored on reasoning rather than on plumbing that day. That's a reasonable design. It also means the eval measures the model's behaviour conditional on correct retrieval, which is a different quantity from the one users experience.

So when retrieval breaks, the eval doesn't just fail to fall. It keeps reporting the model's performance on the well-formed inputs it is still being handed, and any genuine improvement to the prompt shows up as a rise. The metric moves in the reassuring direction while the feature is actively wrong. Penhale's team had a graph trending up on a dashboard for nine days, and the graph was accurate.

The fix is one integration test, and it's unglamorous: assert on what retrieval returned, not on what the model said about it. Given a claim id, does the retrieved set contain that claim's report and nothing belonging to another customer? That assertion doesn't need a model, costs nothing, and would have failed within an hour.

The test: find one eval case and trace where its input came from. If a fixture supplied it, name the test that checks the real path produces the same thing. If there isn't one, that path is untested no matter how many evals you run.

Where the bugs actually are depends on the shape of the feature

Weighting every layer equally is how teams end up with the eval-heavy, integration-light profile that produced the story above. The concentration is predictable from the architecture:

• Retrieval-heavy (RAG). Most defects are layer 2. The model is rarely the weak link; the index, the filter, the chunking and the freshness of the corpus are. Weight integration tests hardest and assert on retrieved sets directly.

• Agentic / tool-using. Layers 2 and 4. Tool execution has side effects, so a wrong call is a wrong action, and an adversarial input can steer it. Test each tool's contract in isolation, then test refusal under pressure.

• Open user input. Layers 4 and 5. Anything the public types at will be probed, and the interesting inputs are the ones nobody on the team would think to write down.

• Single-shot generation over trusted context. Genuinely eval-dominant. This is the one architecture where "we have evals" is close to a strategy — and it's the least common in production.

Notice that the archetype tells you the answer before any incident does. You can do this allocation on a whiteboard the week you pick the architecture.

The test: name your feature's archetype, then compare it to where your tests actually are. If the counts don't match the shape, you're testing the layer that was easiest to test.

When the pyramid is the wrong frame

Layers are a coverage model, not a mandate to build all five:

• A prototype that hasn't found its failure modes yet. Building five layers before you know what breaks encodes a guess as infrastructure. Read outputs, break it by hand, then decide what to automate.

• An internal tool with a human reading every output. The human is layer 5, and a real one. Formalising the rest can cost more than the failures do — say so on the record rather than drifting into it.

• When the layers aren't separable. Some agent frameworks don't let you assert on an intermediate step without rebuilding the harness. Then the honest move is heavier layer 5 and tighter blast radius, not a layer-2 suite you can't actually write.

• When a layer belongs to a vendor. If retrieval is a managed service, your integration test checks the contract you depend on — the fields, the filter semantics, the staleness window — not their internals.

The failure mode this section guards against is a team building all five thinly and having none of them good. Two layers that genuinely execute are worth more than five that gesture.

The test: for each layer you claim to have, name the last real bug it caught. A layer with no catches is either pointed at the wrong thing or protecting something that isn't breaking.

What to change this week

Don't build a layer. Assign one bug.

Take the last defect that reached a user and write down which layer it lived in. Then find the test that should have caught it, and if there is no such test, ask why not — usually the answer is that the layer was never executed by anything, which is a five-minute finding and a much shorter fix than a new test suite.

Then check one eval case's input provenance. Fixture-supplied inputs are fine and normal; the question is only whether anything else exercises the real path that assembles them in production.

Penhale added four integration assertions the week after: retrieval returns the requested claim, returns nothing from another policy holder, fails loudly when the corpus is stale, and matches the fixture the evals use. The eval suite didn't change. The dashboard still trends up — and it now means what the team thought it meant for nine days.