Testing Non-Deterministic Systems Without Losing Your Mind

Teams reach statistical assertions on their own — run it N times, assert the pass rate clears a threshold. Almost nobody says what N has to be, and at the sample sizes teams actually use, a passing run and a failing run are the same measurement.

Illustrative composite: Penhale Mutual is ShiftQuality's running case. Its figures are illustrative, not measured; the probabilities are real arithmetic.

Penhale Mutual's claims-triage assistant reads a first-notice-of-loss report and drafts a summary plus a recommended handling path. Its faithfulness check ran on every commit: 20 samples, assert a pass rate of at least 0.90.

Over one sprint the check went red four times. Each time somebody re-ran it, each time it came back green, and by the end of the sprint the convention was in the runbook: if faithfulness is red, re-run it before you investigate.

Then a prompt revision shipped that genuinely reduced faithfulness. The check went red twice. Both times, somebody re-ran it.

The test was right both times. It had also been right, in exactly the same way, on the four occasions when nothing was wrong, and from inside the test there was no way to tell those apart. The re-run convention was not laziness; it was the rational response to a test whose false-alarm rate the team had measured by hand.

Property assertions fix what you assert, not how many runs you assert it over

The first move away from a flaky AI test is always the same, and it is correct: stop asserting the exact string.

def test_summary():
    summary = generate_summary(complaint)
    assert summary == "Customer received wrong color widget and requests refund."

Assert instead the properties that made you want that string:

def test_summary():
    summary = generate_summary(complaint)
    assert 10 < len(summary.split()) < 30          # length band
    assert "refund" in summary.lower()             # the requested action
    assert any(w in summary.lower()
               for w in ["wrong", "incorrect", "different"])  # the issue

That removes most of the flakiness people complain about: the test stops demanding what the system cannot reliably produce. But notice what it did not decide. Property assertions fix what is asserted, not how many runs it rests on — this post's unit: the number of samples behind a quality claim.

The unit is narrow on purpose. Whether the score you are averaging measures anything is a calibration question; which layer a bug lives in is a coverage question; which failures deserve budget at all is a prioritisation question. All three consume a pass rate. None asks how many runs it came from.

The calibration question is the one this gets confused with most, and the two fail in opposite directions. A biased scorer gives you a smooth, confident, wrong number, and more runs only make you more confident in it. A high-variance measurement gives you a jumpy number, and more runs genuinely fix it. Almost every "flaky AI test" argument is a variance problem being discussed as if it were a correctness problem — which is why the instinct to loosen the assertion, or delete the test, makes things worse.

The test: find one equality assertion on a model output in your suite and rewrite it as three property assertions. Then notice you still have no idea how many times to run it.

Twenty samples against a 0.90 bar goes red one run in three

Some properties cannot be settled in one run. Faithfulness is one: a summary may be faithful 90% of the time on the same input. So you sample:

def test_faithfulness():
    results = [is_faithful(complaint, generate_summary(complaint))
               for _ in range(20)]
    rate = sum(results) / len(results)
    assert rate >= 0.90, f"Faithfulness {rate:.0%} below threshold"

Twenty runs, one threshold, one red-or-green. The 20 is almost always chosen because it felt like enough and calls cost money.

Here is what that test does. If the true faithfulness rate is exactly 0.90, the chance of seeing 17 or fewer successes in 20 samples — what a rate below 0.90 means at n = 20 — is about 32%. Roughly one run in three, a system sitting on its bar fails its own test.

That is the flakiness teams blame on the model, and the diagnosis is wrong in a way that matters. The randomness is in the measurement, not in the system. Penhale's assistant was 90% on every one of those four red runs. What varied was a 20-sample estimate of a 90% rate.

This is arithmetic, not a finding — a binomial calculator reproduces it in a minute. It is also the whole explanation of the re-run convention: a team re-running a red check has inferred, correctly, that red carries almost no information.

The test: grep your suite for the loop count next to a model call. Whatever number you find is the resolution of every quality claim your CI makes.

The regression you care about is smaller than your test can resolve

Resolution improves with the square root of the sample, which means slowly and expensively:

Samples per run	What the run can actually resolve (≈95% interval at a ~90% rate)	Cost per run
20	±13 percentage points — cannot tell 90% from 78%	20 calls
100	±6 points — cannot tell 90% from 84%	100 calls
400	±3 points	400 calls

Those are normal-approximation intervals, the quick formula most people reach for, and at small samples it flatters you. The exact interval for 18 passes in 20 runs reaches down to about 68%, so the honest first row is worse still: twenty samples cannot tell a 90% system from a 70% one.

Read the first row against the failure you are afraid of. A prompt change that quietly moves faithfulness from 92% to 86% is a real, shippable degradation — a slow rise in adjuster escalations rather than an incident. At n = 20 it is invisible; at n = 100 it is marginal. Meanwhile the test goes red regularly when nothing has happened: loud about nothing, silent about the thing it exists to catch.

The repair teams reach for makes it worse. Faced with a check that keeps going red for no reason, somebody lowers the threshold: 0.90 becomes 0.85, the noise stops, the board goes green. The test's poor resolution has been spent buying silence. At n = 20 against a 0.85 bar, the instrument cannot detect anything short of collapse — and it is still called a quality check.

One move buys resolution without buying samples: stop comparing against a constant. Run two arms in the same job — old prompt and new prompt, same inputs, same model — and compare them. Because both see the same inputs, the case-to-case difficulty that dominates the noise cancels. What remains is each arm's run-to-run sampling noise, so a difference larger than that shows up at sample sizes where neither arm's absolute rate is worth reporting.

The test: take your threshold and your n and work out what size regression you could detect. If it is larger than the regression that would embarrass you, the test is decorative.

Move everything exact off the sample budget

Before buying more samples, check how many you are wasting.

Penhale's faithfulness suite scored ten properties. Six turned out to be exact checks wearing statistical clothing: policy number cited, currency formatted, no monetary amount absent from the claim file, valid JSON, length inside the band, required fields present. Each is decidable by code on one output, each was paid for twenty times a commit, and each diluted a pass rate meant to be about something else.

An exact check costs one run; a distributional claim costs N. Anything promoted out of the statistical suite is sample size you can spend where it is needed, and the surviving pass rate sharpens because it measures one kind of thing. This is the split acceptance criteria draw between an invariant and a distributional criterion, cheaper to make there than to discover in CI.

The four survivors moved to a nightly job at 300 samples, and the release bar moved with them: a threshold is only a shipping gate if the run behind it can resolve the difference it enforces. The per-commit job kept its 20 samples but dropped to a 0.70 floor and was renamed for what it does — assert that nothing has collapsed, not that quality is good.

For the wider picture of what quality engineering means once AI is in the product, see Quality Engineering for AI.

That is not a weaker test. At n = 20 you can support "no catastrophic collapse" and you cannot support "faithfulness is at least 90%." Asserting only what you can support is what makes red mean something.

The test: list your scored properties and mark each one a piece of code could decide on a single output. Those are not statistical tests; they are invariants you are paying N times for.

Temperature 0 makes your test reproducible, not your system deterministic

The other escape attempt is to set temperature to 0 in CI and go back to writing normal tests.

It buys one real thing: replayable failures. When a call fails you can run it again and see the same failure — the difference between debugging and guessing. Keep it for wrapper tests, parsers and prompt construction, where you test your code rather than the model.

It does not buy determinism — providers say so plainly; Microsoft's reproducible-output documentation states that "determinism isn't guaranteed" even with a fixed seed — and it costs something. Many production systems run above 0, because outputs are often better there. A suite pinned at 0 then measures a distribution your users never see, and the variance you removed from CI is fully present in production. Nor does it survive a model update or a change of inference stack.

On some newer models the escape hatch is gone entirely. Anthropic's API no longer accepts a temperature setting other than the default for models released after Claude Opus 4.6, so there is nothing to pin. Sample-size discipline is the part of this post that does not depend on a provider setting.

The test: check the temperature production runs at, then the temperature your behaviour tests run at. If they differ, name in one sentence what your tests are therefore not measuring.

When this is the wrong move

Sample-size discipline costs money and attention, and sometimes buys nothing.

• The output is fully checkable by code. A feature that extracts four fields into a schema has invariants and no distributional claim. Assert them, run once, stop.

• You are still prototyping. Before you know what good looks like, 400 calls a night is a bill for confirming a guess. The sample-size question arrives when a threshold starts blocking something.

• Temperature 0 where your provider allows it, pinned model, narrow inputs. Variance really can be near zero, and one run is a fair test — until the pin moves. Treat the pin as the control, and re-measure the day it changes.

• Nobody acts on the number. A statistical test whose red never stops anything is a cost with no decision behind it.

The test: for each statistical test, name the decision its red state changes. If there is no decision, the sample size does not matter, because the test does not matter.

To write the requirement itself, the AI Feature Requirements Worksheet walks one feature through ten moves: hard and soft properties, pass-rate thresholds, failure-mode bounds, drift monitoring and audit logging.

What to change this week

If your team has already switched off a flaky AI test, start there. Don't delete it — count it. Run it thirty times against the current system and you get a rate: a live production property nobody currently knows, and usually more interesting than the test ever was.

Then open your CI config and find the loop count next to a model call. Put it beside your threshold and work out from the table above what regression that pair detects. Then list the properties inside that test and mark the ones code could decide. Two hours, and it reclaims most of the sample budget.

Penhale's faithfulness dashboard now reports two numbers: the nightly rate at 300 samples, and the per-commit floor. Red on the per-commit job gets investigated rather than re-run, because it now happens about as often as something is wrong, and the runbook line is gone. What did not change: the model, the prompt, the assistant's real faithfulness rate. Only the measurement changed, and the measurement was the broken part.

Sources

• NIST/SEMATECH, e-Handbook of Statistical Methods, §7.2.4.1 "Confidence intervals" — the normal-approximation interval for a proportion used in the table, and why it is an inferior approach at small samples. The 32% false-alarm figure is the exact binomial probability P(X ≤ 17) for n = 20, p = 0.90; any binomial calculator reproduces it.

• Microsoft Learn, "How to generate reproducible output with Azure OpenAI" — "Determinism isn't guaranteed," even with a fixed seed and matching parameters.

• Anthropic, Messages API reference: temperature, accessed October 2026 — temperature is deprecated; models released after Claude Opus 4.6 reject values other than the default.

• Penhale Mutual and its CI history are an illustrative composite, not a record of a specific team.

---

Part of the Shipping AI guide — ShiftQuality's complete map to getting AI into production.