Quality Engineering for AI
- Shawn West
- 2 days ago
- 2 min read
Most writing about AI engineering is about building: pick a model, write a prompt, wire up retrieval, ship. This guide is about the other half — the half that decides whether what you shipped is any good. Quality engineering for AI is the discipline of scoping, defining, testing, and gating AI features so that "it works" is a claim you can defend with evidence, not a demo that happened to go well.
It starts from a conviction this site returns to often: projects fail in discovery, not in code — and AI hides that failure longer, because a fluent, confident output looks like success even when it is answering the wrong question. You cannot test your way to a good AI feature you never scoped. So the path below runs in order: scope the judgment first, define what "good" and "done" mean, test the whole system, then gate the release on evidence.
This is a learning path. Read it top to bottom, or jump to the stage you're stuck on.
1. Scope it before you build
Every AI quality problem traces back to a decision nobody wrote down. Before a prompt exists, decide what the output is for, what good looks like on a real hard input, and what bad-but-plausible looks like.
Discovery for AI Features: Scoping Before You Prompt — the five questions to answer before prompting, a one-page scope template, and how your scoping work becomes your first eval set.
2. Define what "good" and "done" mean
AI features break the assumptions acceptance criteria and Definitions of Done were built on: the same input can yield a different output. These two reframe both so they survive non-determinism.
Acceptance Criteria for Non-Deterministic Features — three verifiable criterion shapes (invariants, distributional-over-an-eval-set, boundary) so a story you can't demo becomes a story you can sign off.
Definition of Done for an AI Feature — the team-wide checklist that extends your normal DoD: evals in CI, failure handling, cost, rollback, and sign-off on evidence instead of the demo.
3. Test the whole system
Evals are one layer, not the strategy — and your test time is finite against an infinite input space. These two cover what to test and where to spend.
A Test Strategy for LLM Features (Not Just Evals) — the five-layer test pyramid (unit, integration/retrieval, evals, adversarial, production monitoring) and why the shipped bugs live in the layers evals never touch.
Risk-Based Testing for AI: Where to Spend Your QA Budget — enumerate AI failure modes, score them by likelihood x impact, and point your testing at what a failure would actually cost.
4. Gate the release
The standard becomes real only when the pipeline refuses to ship what doesn't meet it.
Quality Gates for Shipping AI — the four checks an AI release gate needs (eval threshold, invariant suite, cost/latency budget, rollback readiness), and where to place them in the pipeline.
The through-line
Scope the judgment, define it, test the system that produces it, and gate the release on the evidence. Do those four and "is the AI feature good?" stops being a question you argue about after launch and becomes one you answered — on purpose — before it. That is what quality engineering for AI actually is.
