Quarantine Flaky Tests — Test Automation in Practice, Part 9

Illustrative composite: Tolvern Freight's shipping suite, the system in Flaky Tests: Diagnosis and Cure. That article covers which cause you have and how to tell. This builds the machinery that holds a flake safely while you find out — and forces it back out again.

Before you start

You need:

• A test suite in CI with more than one flaky test. With one, you don't need a process; you need an afternoon.

• CI that can store a small artifact per run (JUnit XML is ideal, and almost every runner emits it).

• Permission to make a test non-blocking. If every test must gate the build, quarantine is a policy conversation before it's a technical one.

• Somewhere durable for a list — a checked-in file is fine and better than a wiki.

About 70 minutes. Examples are pytest; the marker mechanism has a direct equivalent in every runner.

What you'll build

A quarantine with three parts: automatic detection from rerun history, a marker carrying an owner and an expiry date, and a weekly report that makes a growing list impossible to ignore.

Step 1: Collect the evidence before writing any code (10 min)

You cannot quarantine what you can't identify, and memory is a poor detector — people nominate the test that annoyed them most recently, not the one that fails most.

Turn on JUnit XML output and keep it:

pytest --junitxml=results/$(date +%Y%m%d-%H%M%S).xml

Then count failures per test ID across the last few weeks of runs:

# scripts/flake_report.py
import glob, collections, xml.etree.ElementTree as ET

fails, runs = collections.Counter(), collections.Counter()
for path in glob.glob("results/*.xml"):
    for case in ET.parse(path).iter("testcase"):
        tid = f"{case.get('classname')}::{case.get('name')}"
        runs[tid] += 1
        if case.find("failure") is not None or case.find("error") is not None:
            fails[tid] += 1

for tid, n in fails.most_common(20):
    print(f"{n:4d}/{runs[tid]:<4d}  {n/runs[tid]:6.1%}  {tid}")

Check: the top of that list should surprise you at least once — a test nobody complains about, failing more than one people do complain about. If the list exactly matches the team's intuition, you probably don't have enough history yet; wait for another week of runs.

Step 2: Set the threshold, and write down why (5 min)

A flake is a test that fails and then passes with no code change. Turning that into a rule needs a number.

Tolvern used ≥2 failures in the last 50 runs, with no correlated code change. Not because two is principled, but because one failure is often a real bug and they wanted the process to be slow to trigger.

Check: apply your threshold to the Step 1 output. It should select a handful, not a third of the suite. If it selects a third, your suite has a systemic problem — an environment or data issue — and quarantining individually will just hide it.

Step 3: Build a marker that carries an expiry (15 min)

The critical design decision. A marker that only says "this is flaky" produces a list that grows forever.

# tests/conftest.py
import datetime, pytest

def pytest_configure(config):
    config.addinivalue_line("markers", "quarantined(owner, until, reason): non-blocking flake")

def pytest_collection_modifyitems(items):
    today = datetime.date.today()
    for item in items:
        m = item.get_closest_marker("quarantined")
        if not m:
            continue
        until = datetime.date.fromisoformat(m.kwargs["until"])
        if until < today:
            item.add_marker(pytest.mark.fail_the_build_expired_quarantine)
        else:
            item.add_marker(pytest.mark.xfail(strict=False, reason=m.kwargs["reason"]))

Applied to the test:

@pytest.mark.quarantined(
    owner="rosa",
    until="2026-09-15",
    reason="fails ~1/20; order dependence suspected, not yet confirmed",
)
def test_carrier_callback_updates_status():
    ...

Three fields, all mandatory. The expiry is the one that makes this a debt instrument rather than a dumping ground — after that date the marker stops protecting the test and starts failing the build.

Check: set until to yesterday on one test and run the suite. The build must fail, and the message must name the test and its owner. If it silently keeps passing, the expiry is decorative and the list will grow.

Step 4: Make quarantined tests still run (5 min)

A quarantined test that is skipped tells you nothing. It must still execute — you just don't let it fail the build.

xfail(strict=False) does exactly this: the test runs, and both outcomes are tolerated. skip would not.

Check: run the suite and find the quarantined test in the output. It should report as xfail or xpass, never skipped. If it's skipped, you've lost the data you need in Step 7 to know when it's fixed.

Step 5: Publish the list where it's uncomfortable (10 min)

# scripts/quarantine_list.py — run in CI, output to the build summary
for item in collect_quarantined():
    days = (item.until - datetime.date.today()).days
    flag = "OVERDUE" if days < 0 else f"{days}d"
    print(f"{flag:>8}  {item.owner:<8}  {item.nodeid}")

Put the output in the build summary or the team channel — somewhere it appears without anyone choosing to look.

Check: the list is visible to someone who didn't write it. A quarantine list that lives only in the code is a list nobody reads, and unread lists grow.

Step 6: The decision point — extend, fix, or delete (10 min)

At expiry, three options are genuinely available, and picking by default is how lists rot.

The signal that decides it: what has this test ever caught?

• It has caught real defects, and you know which → fix it. Diagnose first with the discriminating experiments; do not add a wait and re-quarantine.

• It has never caught anything, and it's been quarantined more than a month → delete it. A test that fails one run in twenty and has never found a bug has negative value: it costs attention every week and buys nothing.

• It's genuinely non-deterministic because the system is → it isn't flaky. Change the assertion to measure the agreed tolerance, and take it out of quarantine as a normal test.

Extending the date is not on this list. One extension, with a reason, is a judgement call; a second is a decision to keep it forever.

Check: for each test in your quarantine right now, name the last real bug it caught. Every test where nobody can name one is a delete candidate today, not at expiry.

Step 7: Prove a fix before releasing it (5 min)

pytest tests/test_webhooks.py::test_carrier_callback --count=50 -q          # fixed order
pytest tests/test_webhooks.py::test_carrier_callback --count=50 --shuffle -q  # shuffled

Check: 50/50 in both modes before the marker comes off. If it passes shuffled but you never ran it shuffled before the fix, you don't know the fix did anything — you may have just moved the ordering.

Step 8: Watch the aggregate, not the individuals (10 min)

Track one number weekly: the size of the quarantine list.

Individual entries will always churn. The trend is the diagnosis:

• Flat and small — the process is working.

• Growing steadily — you are quarantining faster than you are fixing, and the suite is being quietly decommissioned.

• Suddenly spiking — this is rarely many new flakes; it's usually one environmental change. Look at CI runners and shared fixtures before looking at the tests.

Check: you can state today's number and last month's from a report rather than from memory.

The wrong quarantine beside the right one

	Common version	This version
Marker records	"flaky"	owner, expiry, suspected cause
Test still executes	often skipped	yes, xfail
Expiry	none	fails the build when passed
Visibility	in the code	in every build summary
After six months	grows	flat or empty

You're done when

• An expired marker fails the build, naming the test and its owner.

• Quarantined tests report as xfail/xpass, never skipped.

• The list appears somewhere nobody has to opt in to see.

• You can state the list's size this week and last month.

• No entry lacks an owner or a date.

Troubleshooting

Quarantined tests vanish from the report. You used skip instead of xfail. You've lost the signal that tells you when it's fixed.

A quarantined test starts passing consistently. xpass — that's your cue to remove the marker. Confirm with Step 7 first; consistent passing for a week can just be a quiet week.

The list only ever grows. Almost always a missing expiry, or extensions granted by default. Check Step 3 actually fails the build.

Everything got quarantined at once. You have an environment problem wearing a flakiness costume. Un-quarantine all of it and look at what changed in CI.

A "fixed" test comes back weeks later. The original fix was a sleep. Diagnose with the discriminating experiments in the companion article rather than re-quarantining.

Next

Order dependence is the most common cause on that list, and it is really a data-isolation problem — Build a Test Data Factory.