Quality Gates for Shipping AI

A quality gate gets designed as a measurement problem — pick the checks, pick the thresholds, wire them into the pipeline. It is actually a social problem wearing a pipeline's clothes, and it is decided by what happens the first time it says no to someone in a hurry.

Penhale Mutual added an eval threshold to the release pipeline for its claims-triage assistant. Any change to the prompt, the model config or the retrieval settings had to clear 4.0 on the 240-case suite before it could promote. It was well-built, it ran in CI, and it worked exactly as specified.

In its first quarter it blocked 31 of 88 releases. Twenty-six of those 31 were overridden and shipped anyway.

The number in the config was 4.0. The number the organisation actually enforced was 4.0, unless somebody asks — which is not a threshold, it's a formality. And the interesting part wasn't the override count. It was what happened when someone finally read the 26 override reasons in one sitting.

The unit is the moment of promotion

Everything else in testing an AI feature produces information. Eval scores tell you how good the output is. Layers tell you where a bug lives. The register tells you what to worry about. None of them stop anything.

A gate is the one artifact whose job is to refuse. Its unit is a single promotion decision: this build, this moment, yes or no. That makes it the only place the standard becomes binding — which is also why it's the only place that gets argued with. The Definition of Done is the contract; the gate is where the contract is either enforced or quietly isn't.

The test you can run: ask what your pipeline would have done, unattended, with the last AI change that turned out badly. If the honest answer is "promoted it," you have checks, not a gate.

A gate is not a test — it is a refusal that has to survive pressure

The engineering is the easy half. A threshold in a config file is an afternoon. What determines whether you have a gate is what happens at 4pm on a Friday when it says no and a demo is on Monday.

Two failure shapes, and they're opposites:

Too tight, so it gets overridden. The gate blocks on noise, people learn that blocks are usually spurious, and overriding becomes the normal path. Once that norm exists, the gate stops catching real problems too — not because it fails to fire, but because nobody reads a fire alarm that has cried wolf all quarter.

Too loose, so it never blocks. Comfortable, invisible, and it makes everyone feel covered. A gate that has never blocked anything has not been tested any more than a rollback plan nobody has executed.

Both look fine on a dashboard. The distinguishing question is what happened the last time it said no, and the answer lives in your override log rather than your config.

The test: find the last time your gate blocked a release. Ask what happened in the next thirty minutes. If the answer is "we shipped it," write down who approved that and whether their name is recorded anywhere.

Your override rate is the real threshold, and the reasons are the finding

Penhale's 26 overrides had been logged dutifully and never read as a set. Read together, they weren't 26 reasons. They were three, and only one of them was about quality at all.

Override reason	Count	What it actually says
"Eval flaky, passed on rerun"	11	The suite is non-deterministic and nobody trusts it
"Score drop is on cases we don't care about"	9	The threshold is a single average over unlike things
"Time-sensitive, will follow up"	4	The gate is real; the schedule beat it
"Known issue, tracked"	2	Legitimate, and correctly recorded

Twenty of twenty-six were the team routing around a measurement problem rather than around a quality standard. The gate wasn't too strict. It was reporting a number people had good reason to disbelieve — a flaky suite and a single average that mixed a catastrophic-failure slice in with formatting nits, so a drop could mean anything.

That reframes the fix entirely. Tightening enforcement would have made it worse; the honest move was to fix the measurement and then make the gate binding. Penhale did three things: pinned the judge and temperature so reruns matched, split the single average into three named slices with their own thresholds, and made overrides require a second name — not to add friction, but so the log would keep being worth reading.

Overrides in the following quarter: four. Blocks: nine.

The test: read your last quarter's override reasons in one sitting and group them. If most of them are about the measurement rather than about the risk, your gate isn't being ignored — it's being correctly distrusted.

What a gate can actually check

Your existing gate keeps its checks — tests, coverage, security scan. These are the additions that make it see the model, and each has a different failure mode when you get it wrong.

Check	Blocks on	The mistake to avoid
Eval threshold	Score below bar on a versioned set	One average over unlike cases. Use named slices with their own bars
Invariant suite	A single occurrence — leaks, fabricated policy, disallowed content	Averaging it. A zero-tolerance failure must never be a percentage
Cost & latency budget	p95 latency or cost-per-call over budget	Omitting it. A quality win that triples cost is a different incident
Rollback readiness	Kill switch absent or unverified	Verifying it during the incident instead of before promotion

The invariant row is the one that most changes behaviour, and it's where the P1 modes from your register land. It is also the row that should never be overridable at all — if a promotion can proceed past a cross-tenant leak on somebody's say-so, the gate has no floor.

Run the cheap half pre-merge, where a regression is a pull-request comment. Run the full set pre-promotion, where the cost and rollback checks make sense.

The test: for each of the four rows, name the last build it blocked. Any row with no blocks ever is either mis-tuned or is checking something that doesn't vary.

When a gate is the wrong instrument

• Before the measurement is trustworthy. A gate built on a metric nobody believes manufactures overrides, and the override habit outlives the fix. Calibrate first, gate second.

• When the risk is genuinely novel. A gate checks what it was told to check. A new adversarial surface passes green, which is why high-cost-of-wrong releases still want a named human looking — encode the sign-off rather than pretending judgment isn't involved.

• Internal tools with a human in the loop on every output. The reviewer is the gate. Adding a pipeline one duplicates it and trains people to click past both.

• Exploratory or spike work. Gating a branch nobody will ship costs time and teaches the team that the gate is theatre.

• When the real constraint is elsewhere. If releases are slow because of review queues, a stricter gate adds delay without adding safety, and it will be blamed for the delay.

The test: ask what specific failure the gate has prevented since you built it. If you can't name one, it's either well-tuned and lucky or it isn't doing anything — and the override log will tell you which.

What to change this week

Don't tune the threshold. Read the overrides.

Pull every override from the last quarter, put the reasons in one document, and group them. You are looking for the reasons that are about the instrument rather than the risk — flaky, rerun, doesn't apply to us, that slice doesn't matter. Those are a specification for what to fix, written by the people who work around your gate every week.

Then split any single averaged threshold into named slices, and pull your zero-tolerance checks out of the average entirely so a rare severe failure can never be smoothed into an acceptable number.

Penhale's gate blocks about one release in ten now and gets overridden roughly once a month, with two names on each. Nobody has proposed removing it, which is the most reliable signal that it's set correctly — a gate people fight to keep is one they believe.