Flaky Tests: Diagnosis and Cure

Every team knows how to fix the five causes of flakiness. Almost none can tell you which one they have. That gap is the whole problem — and it's a measurement problem, not a fixing one.

Tolvern Freight had a carrier-webhook test that failed roughly one run in twenty. Somebody added a 500 ms sleep before the assertion. The failure rate dropped to about one in fifty, the build went quiet, and the team moved on.

Six weeks later it came back at one in ten, on a slower CI runner. This time someone ran the suite with the test order shuffled, and it failed on the first attempt, every time. The cause had never been timing. An earlier test in the file left a stale carrier account in the database, and the webhook test passed only when it happened to run before it.

The sleep had not fixed anything. It had made the failure rarer, which made it harder to reproduce, which is the opposite of progress. A change that lowers the failure rate without identifying the cause is strictly worse than no change, because the failure rate was the only evidence you had.

You cannot fix what you cannot reproduce on demand

The instinct with a flake is to reach for the fix — a wait, a retry, a beforeEach cleanup. Each of those is a plausible cure for a different disease, and applying one before diagnosis has two costs. It may not work. And when it partially works, it removes the signal you needed.

The discipline that changes outcomes is dull: before any fix, run an experiment that distinguishes between causes. There are five common causes, and each one is separated by a cheap, specific manipulation — usually a single command-line flag.

Suspected cause	The discriminating experiment	If it now fails consistently	If it's unchanged
Order dependence / shared state	Run the suite with -p no:randomly off, or --shuffle on — reverse the order	Order dependence confirmed. Something upstream is leaking.	Not order
Concurrency	Run with one worker: -n 0 / --runInBand	Parallelism is the cause; two tests share a resource	Not parallelism
Timing / async	Run on a deliberately throttled machine, or add CPU load	A real race — the code is waiting on the wrong thing	Not timing
Clock / date	Pin the clock (freezegun, sinon.useFakeTimers) to the failure time	Time-dependent — a boundary, a timezone, a TTL	Not the clock
External dependency	Run with the network blocked or the stub forced	The test reaches something real it shouldn't	Not external

Each row takes under a minute and eliminates a whole class. Two or three of them will usually leave exactly one candidate standing.

Tolvern's team ran row 3 first, got an inconclusive result, and reached for the sleep anyway. Row 1 would have taken ten seconds and pointed straight at the cause.

The test you can run: for the flaky test currently annoying you most, name the experiment that would rule out your leading theory. If you can't name one, you don't have a theory — you have a preference.

Rerun-to-fail is the measurement, and most teams don't have it

The experiments above need a baseline: how often does this test fail, under what conditions? Without a number, "it seems better" is the only available verdict, and it's the verdict that let Tolvern's sleep survive six weeks.

The cheapest useful instrument is a loop:

pytest tests/test_webhooks.py::test_carrier_callback --count=50 -p no:randomly -q

Fifty runs, fixed order. Then fifty more with shuffling on. The difference between those two numbers is the diagnosis for the most common cause, and it costs a coffee's worth of waiting.

The organisational version is the same thing over time: record every test failure with its test ID, and count distinct failures per test per week. A test that fails in runs that later pass without a code change is flaky by definition — you don't need a person to label it.

The test: ask your CI whether it can tell you which test failed most often last month. If the answer requires someone to remember, you have no flake data, and every conversation about flakiness will be about impressions.

Fix the cause you found, not the cause you expected

Once the experiment names the class, the fixes are unglamorous and well known:

• Order dependence — the leak is almost always something not rolled back. Move to transaction-per-test rather than adding cleanup calls; cleanup is a list you will forget to extend. This is a test data management decision more than a flakiness one.

• Concurrency — find the shared resource. It is usually a fixed port, a fixed filename, a shared account, or a sequence. Make it unique per worker rather than serialising the suite.

• Timing — replace the sleep with a wait-for-condition on the thing you actually care about. wait_until(lambda: booking.status == "confirmed", timeout=5) fails in five seconds with a useful message; sleep(0.5) fails randomly with none.

• Clock — pin it. Then test the boundary deliberately, since a test that only fails at month-end has found a real bug worth keeping.

• External — stub it at the boundary you own. If the test genuinely must hit a real service, it isn't a unit or integration test any more and shouldn't run on every commit.

The only one of these that overlaps with the instinctive fix is the timing row — and even there, the correct fix is a condition, not a duration.

The test: for each flake you've fixed in the last quarter, can you name which of the five classes it was? If several come back as "we added a wait and it stopped", those are unresolved and will return on a slower runner.

Quarantine is a debt instrument, and it needs a maturity date

A test that fails randomly and stays in the main suite trains people to rerun red builds, which costs you every other test at the same time. Quarantine — moving it out of the blocking path — is the right immediate move.

It is also where flaky tests go to live forever. A quarantine with no exit is just a slower delete, with the added cost that the test still runs, still takes time, and still shows up in reports as coverage you don't have.

Three things make it a debt instrument rather than a dumping ground:

• An owner, named at the moment of quarantine, not assigned later.

• A date, after which the test is either fixed or deleted — genuinely deleted, not re-quarantined.

• A visible count. If the quarantine list has grown every month for six months, the problem is not the tests.

Deleting a flaky test is a legitimate outcome and is often the right one. A test that has never caught a real defect and fails one run in twenty has negative value — it costs attention every week and buys nothing. The uncomfortable question is whether you can tell the difference, which requires knowing what the test has ever caught.

The test: for every test in quarantine right now, name the last real bug it caught. Any test where nobody can name one, and which has been quarantined more than a month, should be deleted this week.

When the flake is the only honest signal you have

Sometimes the test is right and the system is genuinely non-deterministic. Distributed systems, eventual consistency, anything with a queue or a retry — these produce tests that fail occasionally because the behaviour occasionally differs.

Suppressing that with a retry is the most expensive mistake in this whole area, because the test was reporting a real property of production. The signal to watch for: the failure is intermittent but the failure mode is always the same. Genuine flakes tend to fail in scattered ways — a timeout here, a missing row there. A system that is truly non-deterministic fails the same assertion, in the same way, at a stable rate.

When you see that pattern, the fix is not in the test. Either the system needs to be made deterministic at that boundary, or the requirement needs to state a tolerance — which turns a flaky test into a proper one measuring an agreed rate.

The test: collect the last ten failures of the test. If they're all the same assertion failing the same way, stop treating it as flaky and go read the code it's exercising.

What to do this week

Take your worst flake. Don't fix it. Run it fifty times in fixed order, then fifty shuffled, and write both numbers down. That single comparison resolves the most common cause and costs about ten minutes of waiting.

Whatever it tells you, you'll have something the sleep never gave Tolvern: a number that will change in a specific direction when you fix the right thing, and won't move when you fix the wrong one.

The mechanics of quarantining — detection, marking, tracking, and the exit criteria that stop the list growing — are in the companion tutorial, Quarantine Flaky Tests.