top of page

A Test Strategy for LLM Features (Not Just Evals)

  • Shawn West
  • 3 days ago
  • 5 min read

"We have evals, so we're covered." The team said it with confidence, and they'd earned some — the eval set was good, the scores were high. Then the feature shipped and broke in a way the evals had never looked at: the retrieval step was pulling the wrong document, so the model was faithfully summarizing the wrong customer's data. The evals scored the model's reasoning on the text it was handed, and it reasoned fine. Nobody tested whether it was handed the right text. The bug wasn't in the model. It was in the plumbing around the model — and "we have evals" had quietly become "we tested one layer and assumed it was all of them."

Evals are essential, and they are not a test strategy. They're one layer of several, and each layer catches bugs the others can't see. An LLM feature is a system: deterministic glue code, a retrieval or tool-calling step, the model itself, an adversarial surface, and live production behavior — and a test that only exercises the model leaves four layers dark. This is the full test pyramid for an LLM feature: what each layer tests, what it catches, and why skipping any of them ships the bug that "we have evals" can't see. It builds on the distributional criteria from Acceptance Criteria for Non-Deterministic Features — evals are how you verify those — and extends them to the rest of the system.

The five layers

Think of it as a pyramid: cheap, fast, deterministic tests at the base; expensive, judgment-heavy checks toward the top. You want most of your test count at the bottom and the right kind of test at each level — not everything shoved into the eval layer because that's the AI-shaped one.

Layer 1 — Deterministic unit tests (the base)

Most of an LLM feature isn't the model. It's parsing, validation, prompt assembly, output handling, retries, fallbacks. All of it is ordinary deterministic code, and all of it should have ordinary fast unit tests.

- Prompt template renders with the right variables interpolated
- Malformed model output (invalid JSON) is caught and triggers the fallback path
- Token-limit truncation logic keeps the system prompt and trims the right section
- A model timeout returns the graceful fallback, not an unhandled exception

These catch the boring bugs that cause a shocking share of AI incidents. They're cheap; write a lot of them.

Layer 2 — Integration tests (retrieval, tools, APIs)

This is the layer the opening story skipped. Test the glue between your code and the model, and between the model and everything it reaches — retrieval, tool calls, external APIs.

- Retrieval returns the correct documents for a known query (the wrong-customer bug lives here)
- The tool the model called actually executed with the arguments it was given
- Retrieval scoping enforces per-user access (never crosses tenant boundaries)
- A downstream API failure degrades gracefully instead of hanging the response

For RAG especially, retrieval quality often matters more than the model — a perfect model reasoning over the wrong context is confidently wrong. Test retrieval as its own thing.

Layer 3 — Evals (model behavior)

Now the layer teams start with. Evals test the model's judgment on a curated set with known-good expectations: is the output correct, on the inputs we chose, including the hard ones. This is where your distributional acceptance criteria get verified, and it belongs in CI with a threshold.

Evals answer "is the model good at its job?" — a real and necessary question. They do not answer "did it get the right input?" (layer 2) or "does the surrounding code handle its output?" (layer 1). Keep the question scoped and you'll stop expecting evals to catch plumbing bugs.

Layer 4 — Adversarial / red-team

Users, and attackers, will send inputs you didn't curate. This layer probes the failure surface on purpose.

- Prompt injection ("ignore previous instructions…") does not override system rules
- Out-of-scope questions get a refusal or handoff, not a confident fabrication
- Attempts to extract the system prompt or other users' data fail
- Nonsense, empty, and enormous inputs are handled without crashing or leaking

Evals test whether the feature does its job. Red-teaming tests whether it can be made to do something else. Both matter; they're different tests.

Layer 5 — Production monitoring (the test that never stops)

The one layer that runs after ship. Because the same input can drift as models update and inputs shift, testing doesn't end at release — you monitor live outputs, sample them against the rubric, watch cost and latency, and capture failures back into the eval set.

- Sample N production outputs/day scored against the rubric; alert on drops
- Track cost-per-call and p95 latency with alerts on regression
- Log inputs+outputs so a failure can be captured and replayed as a new eval case

Production is where you discover the inputs you never imagined. A feature you stop testing at release is a feature you've stopped knowing the quality of.

Allocating effort

You don't test every layer equally. Match effort to where this feature's risk concentrates:

  • RAG / retrieval-heavy → weight layer 2. The model is rarely the weakest link; retrieval is.

  • Agentic / tool-using → layers 2 and 4. Tool execution and adversarial control of actions are the danger.

  • User-facing / open input → layer 4 and 5. Anything the public can type at, assume someone will attack and surprise.

  • Internal, low-stakes → layers 1–3 may be enough; scale down honestly.

The mistake isn't under-testing one layer. It's testing one layer and calling it the strategy.

Key Takeaway

"We have evals" tests one layer and assumes it's all of them — which is how a retrieval bug ships a confidently-wrong summary of the wrong customer's data while the eval scores stay green. An LLM feature is a system with five test layers: deterministic unit tests for the glue code (the cheap base), integration tests for retrieval/tools/APIs (where the plumbing bugs live), evals for model judgment (verifying your distributional criteria), adversarial/red-team for the attack surface, and production monitoring for the drift that never stops. Each layer catches what the others can't. Weight them toward where the feature's risk actually sits — retrieval for RAG, tools and adversary for agents, red-team and monitoring for anything user-facing. Evals are the layer everyone starts with and the one most mistaken for the whole. The wrong-customer bug wasn't a model failure. It was a missing layer-2 test, hidden behind a high eval score.

bottom of page