End-to-End Testing Without the Pain

The pain of end-to-end testing isn't that the tests are slow. It's that they're flaky — and a suite that fails at random trains the whole team to stop believing red. Once a red build means "run it again," a real regression walks straight through it. Here's the mechanism that makes E2E tests non-deterministic, one flaky test taken from broken to stable, and the single number that tells you how bad your own suite really is.

Here's the failure that actually costs you, and it isn't the one in the tutorials. A team has forty end-to-end tests. Two or three of them fail on most runs — never the same two or three. Everybody knows this. The unwritten rule is: if the build is red, hit "Re-run failed jobs," and if it goes green, merge. The suite has become a coin you flip until it lands heads.

Then one Tuesday a genuine regression ships — checkout silently drops the discount code — and when someone digs into why the E2E suite didn't catch it, the git history is damning: the checkout test did fail on the PR that introduced the bug. Someone re-ran it, it passed on the second try for an unrelated timing reason, and the green checkmark waved the change through. The test worked. The team had been trained not to believe it.

That's the real pain of E2E, and it has almost nothing to do with speed. A slow suite is annoying. A flaky suite is dangerous, because it destroys the one thing a test exists to provide: a signal you trust. Everything else in this guide — selectors, waits, data, what to push down the pyramid — is worth doing only because each one removes a specific source of that flakiness. Flakiness is the disease. The practices are named cures.

What "flaky" actually means, mechanically

A flaky test is one that passes and fails on the same code. That definition matters because it tells you exactly what you're hunting: non-determinism. Somewhere in the test, the outcome depends on something other than the code under test — and that something varies between runs.

For end-to-end tests there are four usual sources, and they're worth understanding as cause-and-effect rather than as a list to nod at.

Timing and implicit waits. An E2E test drives a real browser against a real, asynchronous system. The click returns instantly; the effect — a row appears, a spinner clears, a URL changes — lands whenever the network and the render happen to land. If the test asserts before the effect arrives, it fails. The moment you paper over that with a fixed sleep(2), you've built a race: two seconds is too long on a fast run (wasted minutes) and too short on a slow one (a failure). The test now depends on machine load, which varies. That's the mechanism of the single most common flake.

Shared state. Tests that read and write the same records in a shared environment are coupled through data even when they never call each other. Test A creates a user; test B counts users and asserts "3"; a week later there are four users and B fails — not because B's feature broke, but because the world it assumed drifted. The assertion depends on global state, and global state changes.

Ordering. The moment one test depends on another running first — test 2 logs in and expects the account test 1 created — you've made the outcome depend on execution order. Run them in parallel, or let the framework shuffle them, and the dependency breaks intermittently. Order is not something you control reliably across CI shards, so anything that depends on it is non-deterministic by construction.

Brittle selectors. A selector tied to styling (.btn-primary-v2) or DOM shape (div > div:nth-child(3) > button) depends on things that change for reasons unrelated to behavior. A designer renames a class, the test goes red, and nothing about the user journey actually broke. The signal now fires on refactors, so people learn to ignore it.

Notice the shared shape. In every case the test's result depends on something other than the behavior it claims to verify — the clock, the data, the order, the markup. That dependency is the flake. Each cure below is just the removal of one of those dependencies, and it's worth being able to say which one a given practice removes, because that's how you know whether it's worth its cost.

One flaky test, from red to stable

(Developed example — composite scenario.)

Take the checkout test from the opening, the one that let the discount bug through. Here is roughly what it looked like — a real-ish Playwright test in the shape teams actually write first:

def test_checkout_applies_discount(page):
    page.goto("/products")
    page.click(".product-card:first-child .add-to-cart")
    page.click("#cart-icon")
    page.click(".checkout-btn")

    page.fill("#promo", "SAVE10")
    page.click("#apply-promo")
    time.sleep(1)  # wait for discount to apply

    total = page.inner_text(".order-total")
    assert total == "$45.00"

This test failed roughly one run in five. Before touching it, the team did the one thing that turns "it's flaky" into a diagnosis: they ran it in a loop and read the failures.

# run the single test 30 times, capture pass/fail
for i in $(seq 1 30); do
  pytest tests/e2e/test_checkout.py::test_checkout_applies_discount -q \
    >> flake.log 2>&1 && echo PASS || echo FAIL
done | sort | uniq -c
#   24 PASS
#    6 FAIL

Six failures in thirty — a ~20% flake rate on one test. The Playwright trace on the failing runs showed the actual race: #apply-promo fires an async call to the pricing service; on a slow run that call hadn't returned when the time.sleep(1) expired, so .order-total still read the pre-discount $50.00. The sleep was a bet on latency, and one run in five it lost. Worse, the same mechanism meant that when the discount genuinely broke, the test could still pass — if the render happened to be slow enough that the assertion read a stale value that matched by coincidence, or if a retry landed on a good run. A test that depends on timing can't reliably catch a bug in the thing it's timing.

Two changes stabilized it. Replace every implicit wait with an assertion on the state you actually care about, and pin the selectors to intent:

def test_checkout_applies_discount(page):
    # unique cart per run — no shared state, safe in parallel
    page.goto("/products")
    page.get_by_test_id("add-to-cart").first.click()
    page.get_by_test_id("cart-icon").click()
    page.get_by_role("button", name="Checkout").click()

    page.get_by_test_id("promo-input").fill("SAVE10")
    page.get_by_test_id("apply-promo").click()

    # wait for the OUTCOME, not the clock: assert the settled value.
    # Playwright auto-waits and retries this assertion until it holds or times out.
    expect(page.get_by_test_id("order-total")).to_have_text("$45.00")

The expect(...).to_have_text(...) doesn't sleep — it polls the DOM until the total settles or the timeout expires. On a fast run it passes in 40ms; on a slow run it waits as long as it genuinely needs. The race is gone because the test now depends on the observable outcome (the discounted total appeared) instead of on a guess about how long that outcome takes. Re-run the thirty-loop: thirty passes. And now the test does its real job — if the discount logic breaks, the total never becomes $45.00, the assertion times out, and the build goes red for a reason you can trust.

The general move is the one worth keeping: never wait for time; wait for a condition. A fixed sleep encodes an assumption about latency, and any assumption about latency is a race. An assertion on state encodes the thing you actually mean.

Measure your flake rate — the number that runs this whole decision

You cannot manage flakiness you haven't measured, and most teams have never put a number on theirs. They have a feeling ("the suite's a bit flaky") — which is exactly the fog that lets a real regression hide. Replace the feeling with a rate.

Flake rate, defined observably: of the runs that failed and were then re-run on identical code, what fraction passed on the retry? A failure that passes on retry without a code change is, by definition, a flake.

You can pull this without new tooling if your CI records retries. The cheap version, per test:

flake_rate(test) = (# runs that failed then passed on retry, no code change)
                   ---------------------------------------------------------
                    (total runs of that test)

Three ways to get the number, cheapest first:

• Turn on one retry with reporting and read it. Playwright's --retries=1 marks any test that failed-then-passed as flaky in its report; CI dashboards surface a "flaky tests" count. That count, divided by runs, is your rate. (One retry to measure is fine. Retrying to hide is the trap — more on that below.)

• Rank the offenders. You don't need to fix flakiness in general; you need to fix the three tests causing most of it. Sort tests by failed-then-passed count over the last two weeks. Flakiness is almost always concentrated — a handful of tests generate the majority of the noise. Those are your worst offenders, and they're where the whole payoff is.

• Loop the suspects locally. For each top offender, run it 20–30 times in a loop (the for loop above) and read the traces on the failures. The loop turns "sometimes fails" into a specific race or a specific shared-state assumption you can actually fix.

Then quarantine, don't ignore. The instant a test's flake rate crosses a line your team sets — a common one is any test that flakes more than ~1% of runs — pull it out of the blocking suite into a quarantine group that still runs and still reports but does not gate merges. This is the move that breaks the doom loop: the blocking suite goes back to meaning something (green is trustworthy again), while the quarantined test stays visible as debt with an owner and a due date, not silently retried into oblivion. A quarantine that becomes a graveyard is its own failure mode — put a cap on it (say, no test sits quarantined longer than a sprint) so "quarantine" doesn't just become a slower way to delete your coverage.

The number to pull this week: for your top suite, count failed-then-passed-on-retry over the last 10–20 runs, divide by total runs, and rank the tests by how many of those flakes each one owns. If your suite-level flake rate is above a few percent, your red builds are already being ignored — you just hadn't measured it. The top three tests on that ranked list are your entire near-term backlog.

The retry trap — where the discount bug actually shipped

There's a tempting shortcut that makes all of this look solved: retry failed tests automatically, and count any test that passes "after retry" as a pass. The dashboard goes green. The pain goes away. And this is precisely how the discount regression shipped.

The mechanism is worth stating plainly, because it's the whole reason flakiness is dangerous rather than merely annoying. A test that is genuinely catching an intermittent bug — a race in your code, not the test's — looks identical, on a single run, to a test that is merely flaky. Both are "red sometimes." If your policy is "retry until green, then merge," you will retry the real-bug-catcher until it happens to pass, and ship the bug. Automatic retry-to-green doesn't remove flakiness; it converts your flaky tests into blind spots and hides real defects in the same motion.

The discipline that keeps retries honest: retries may surface flake (run with one retry, log every failed-then-passed as a flaky event, feed it to the rate above) but must never launder it (a test that only passes on retry is not a pass — it's a defect, either in the test or in the code, and it goes to quarantine with an owner). The difference between those two policies is the difference between a suite that measures its own reliability and one that lies to you cheerfully.

What belongs in E2E — and what you're paying for when it doesn't

Every flaky E2E test has a hidden question behind it: did this behavior even belong at the E2E level? Most flakiness is concentrated in tests that are verifying something a cheaper, more deterministic test could have verified without a browser.

The tradeoff is real and worth naming rather than sloganizing. E2E tests buy you something no lower test can: proof that the real, assembled system — real browser, real services, real network — carries a user journey end to end, catching the integration seams where every component works alone but they don't fit together. What you pay for that proof is non-determinism. Every real dependency an E2E test touches is another clock, another shared datastore, another service that can be slow — another source of flake. So the governing principle is: test each behavior at the lowest level that can actually catch its failure, and spend the expensive, flake-prone E2E budget only on the journeys where the assembly itself is the thing you're verifying.

Behavior	Test it here	Why not E2E
A validation rule ("reject negative quantity")	Unit	No integration risk; a browser adds only flake and minutes
Two services agree on a contract	Integration	Deterministic, no UI clock; catches the seam directly
The critical revenue journey (add-to-cart → pay → confirmation)	E2E	The assembly is the risk; nothing lower proves it
Every error message and empty state	Component / unit	Combinatorial at E2E; cheap and stable one level down

The pattern to distrust: a behavior tested at unit and integration and E2E "for extra confidence." That's usually not confidence — it's a tell that the lower tests aren't trusted, and the fix is to make them trustworthy, not to pile a flaky browser test on top. Where each behavior belongs is exactly the argument of the test pyramid versus test trophy; the reason integration tests can absorb so much of what teams reflexively push to E2E is covered here. And the small, fast set of E2E checks you do keep in the blocking path is really a smoke test — the five-minute quality gate: a handful of critical journeys, kept ruthlessly stable, run on every PR, with the full slow suite pushed to merge or nightly.

When E2E isn't worth it at all — the exception. For some systems the cost-benefit genuinely doesn't clear. A backend service with no UI is fully served by integration tests; there's no assembled user journey for E2E to add. A UI that changes weekly by design will spend more engineering time repairing selectors than the tests return. And no automated E2E suite replaces a human noticing that the flow feels wrong — that's the domain of exploratory testing, a discipline not a hack. Write E2E tests where the assembly is the risk and the journey is stable enough to be worth protecting — not because a suite is expected to have them.

What to do this week

Pick your most-run E2E suite and measure its flake rate: over the last 10–20 runs, count the tests that failed and then passed on retry without a code change, divide by total runs, and rank the tests by how many of those flakes each one owns. That single number tells you whether your red builds already mean nothing — and the top three on the ranked list are your whole starting backlog. Loop each of those three 20–30 times locally, read the trace on the failures, and you'll almost always find one of the four causes: a sleep standing in for a state assertion, a shared record two tests fight over, an order dependency, or a selector tied to markup instead of intent. Fix those three, quarantine anything you can't fix this sprint, and your suite starts telling the truth again — which is the only reason to run it.

---

Related on ShiftQuality: where each behavior belongs (Test Pyramid vs. Test Trophy); what to push down from E2E (Integration Testing: When, How, and Why); the small stable set you keep in the blocking path (Smoke Testing: The Five-Minute Quality Gate); and the human check no suite replaces (Exploratory Testing: A Discipline, Not a Hack).

Keep learning. This article is part of the Software Testing Foundations path in the ShiftQuality Learning Center.