Testing the Hard Parts: Async, External Dependencies, and State

The test passed forty times on my laptop. It failed once in five runs in CI, and never the same run twice. Someone added @retry to the pipeline, the build went green, and we moved on — until the same test caught a real ordering bug two months later that the retry had been quietly swallowing the whole time.

The test looked like this:

def test_order_confirmation_sent():
    order = place_order(cart, customer)
    time.sleep(2)                      # let the email go out
    assert emailer.last_sent().to == customer.email

place_order kicked off the confirmation email on a background task. The sleep(2) was a guess about how long that task takes. On a warm laptop, two seconds was plenty. On a loaded CI runner, the background worker sometimes hadn't picked up the task yet when the assertion ran — so last_sent() returned the previous test's email, or None, and the assertion failed. Two seconds wasn't a wait. It was a bet on the scheduler, and CI was a table where the odds were worse.

The previous article in the Software Testing Foundations path was about writing tests that catch real bugs instead of decorating the code. This one is about the three places that intent goes to die: async operations, external dependencies, and stateful workflows. Each resists a straightforward unit test. Each fails in a way that looks like flakiness and is actually a test depending on something it never named. And each has a technique that cracks it once you name the thing.

The one question that unlocks all three

Before any technique, there's a diagnostic move, and it's the same for all three hard parts. When a test is flaky, don't ask "how do I make it pass." Ask:

What is this test actually depending on that varies between runs?

Run the failing test through that question and the answer is never "randomness." It's specific:

• The sleep(2) test depends on the clock — specifically on the background worker finishing inside an interval you guessed at.

• A test that calls a live payment sandbox depends on the network — on a third party's uptime and latency, neither of which is your code.

• A test that fires two operations at a shared record depends on the order they happen to interleave in — which the scheduler decides, not you.

The clock, the network, the order. Every hard-parts flake traces back to one of these three uncontrolled inputs leaking into the test. The fix is never "wait longer" or "retry more." It's remove the dependency — pin the clock to a condition, replace the network with a double, and force the order instead of hoping for one. Name the varying input, then take it away. That's the whole discipline; the rest of this article is the three specific ways to do it.

Takeaway you can run today: open your flakiest test and finish this sentence out loud — "this test passes or fails depending on ______." If the blank is a duration, a remote service, or a race, you've found your fix in the sections below.

Async: wait for a condition, never for a duration

Timing bugs are the ones synchronous tests literally cannot reproduce: a missed await, a callback that fires after the test function has already returned, two coroutines racing on the same field. The code is correct in a single-threaded reading and wrong the moment real scheduling gets involved.

The cardinal rule: never assert after an arbitrary delay. sleep(2) is a prayer, not a synchronization primitive. It fails in both directions — too short and the test flakes, too long and you've added two dead seconds to every run, which across a suite is how a five-minute build becomes twenty. It couples your test's reliability to a machine's current load, the one variable you have the least control over in CI.

The replacement is to wait for the condition you actually care about, not for time to pass:

• For a promise or task you hold a handle to — await it and assert on the resolved value. You're not guessing when it's done; the runtime tells you.

• For a background effect you don't hold — poll for the observable state with a bounded timeout, so you wait exactly as long as needed and then fail fast with a real error instead of hanging.

• For callback or event code — capture the callback with a test double, trigger it yourself, and assert it was called with the arguments you expect. Don't wait for the event; cause it.

The polling helper is worth keeping in every codebase:

def wait_until(predicate, timeout=5.0, interval=0.05):
    """Poll until predicate() is truthy, or fail with a real message."""
    deadline = time.monotonic() + timeout
    while time.monotonic() < deadline:
        if predicate():
            return
        time.sleep(interval)
    raise AssertionError(f"condition not met within {timeout}s")

Now the opening test stops betting on the scheduler:

def test_order_confirmation_sent():
    order = place_order(cart, customer)
    wait_until(lambda: emailer.last_sent() is not None)
    assert emailer.last_sent().to == customer.email

This waits the 40 milliseconds the worker actually needs on a fast machine and up to five seconds on a slow one, then either passes or fails with condition not met within 5.0s — a message that says "the email never went out," which is a bug, not a coin flip.

Takeaway you can run today: grep your test suite for sleep(. Every hit is a timing dependency you can name and replace with a wait_until on the actual post-condition. Do the noisiest one first.

External dependencies: replace the network with a double you control

Your code calls a payment API, a geocoder, a database, an email service. In a unit test, calling those for real is slow, flaky, sometimes expensive, and non-deterministic. When a test fails because Stripe's sandbox is having a bad morning, that test just measured Stripe's uptime, not your code's correctness. You've paid for a test and gotten a status page.

The fix is a test double — an object that stands in for the real dependency. But "mock it" is not a strategy, because the word "mock" hides a real choice about which kind of double, and the wrong kind gives you a test that passes while the system is broken. Getting this choice right is its own topic, covered in Mocking, Stubbing, and Faking: When to Use Each; the short version for the hard parts:

• A stub returns canned answers. Good when you only need the dependency to say something so your code can proceed — "the geocoder returns these coordinates."

• A fake is a lightweight working implementation. Good when you need to verify a sequence or an effect — an in-memory emailer that records what it was asked to send, an in-memory store that actually holds rows.

• A mock asserts on the interaction itself — that a method was called, with these arguments, this many times.

For the confirmation email, a fake is the right tool, because we care that the email was sent correctly, not merely that some method was poked:

class FakeEmailer:
    def __init__(self):
        self.sent = []                 # the recording

    def send(self, to, subject, body):
        self.sent.append({"to": to, "subject": subject, "body": body})

    def last_sent(self):
        return self.sent[-1] if self.sent else None

The sent list is the whole point: it turns an invisible side effect crossing the network into an in-process fact you can assert against, instantly and deterministically. The test now depends on your list, not on someone else's SMTP server:

def test_confirmation_addressed_to_customer():
    emailer = FakeEmailer()
    place_order(cart, customer, emailer=emailer)
    wait_until(lambda: emailer.last_sent() is not None)
    sent = emailer.last_sent()
    assert sent["to"] == customer.email
    assert "confirmation" in sent["subject"].lower()
    assert len(emailer.sent) == 1      # exactly one — no duplicate

The one honest risk with any double: it can drift from the real thing until your test passes against a fake that no longer matches production. That's what contract tests and a periodic integration test against the real service guard against — the subject of Integration Testing: When, How, and Why. Fakes make your unit tests fast and deterministic; one integration test keeps the fake honest.

Takeaway you can run today: find the one external call in your code that fails a test most often. Wrap it behind an interface you can inject, and give it a fake that records what it was asked to do. Assert against the recording.

State: provoke the transition, don't hope for it

The third hard part is the sneakiest, because the code passes every test you write for it in isolation and still corrupts data in production. Stateful workflows — an order moving pending → confirmed → shipped, an inventory count, a booking that can't be double-sold — break when two operations touch the same state at the same time and the transitions interleave in an order you never tested.

The mistake is testing state changes one clean call at a time. That verifies the state machine's logic but never its concurrency. The discovery question points straight at the gap: this workflow depends on the order operations arrive in, and a one-call-at-a-time test has quietly fixed that order to "one at a time" — the single case that never happens under load.

So provoke the race on purpose. Fire the concurrent operations, let them interleave, and assert the state ended up consistent — not just "changed," but "changed to exactly one legal outcome, once":

def test_confirm_is_idempotent_under_concurrency():
    order = place_order(cart, customer, emailer=FakeEmailer())
    # Two workers both try to confirm the same order at once.
    with ThreadPoolExecutor(max_workers=2) as pool:
        results = [pool.submit(confirm_order, order.id) for _ in range(2)]
        [r.result() for r in results]

    final = get_order(order.id)
    assert final.status == "confirmed"          # not "confirmed" twice
    assert count_emails_for(order.id) == 1      # exactly one confirmation

Run this against a naive confirm_order that reads the status, checks it, then writes — and it fails: both threads read pending, both decide to confirm, both send an email, and the customer gets billed-and-emailed twice. That double-confirm is invisible to any single-call test and obvious to this one. The test doesn't just check the happy path; it manufactures the exact interleaving production will eventually hit and pins the invariant that must survive it: one legal end state, one email.

The fix in the code is a real transition guard — a conditional update (UPDATE orders SET status='confirmed' WHERE id=? AND status='pending') or a lock — so the second confirm is a no-op. But you can't verify that guard exists without a test that creates the collision. This is the discovery move made concrete: the varying input was the interleaving order, and instead of hoping for a benign one, the test forces the worst one every run.

Takeaway you can run today: pick one state transition that must happen at most once (confirm, charge, ship, redeem). Write a test that fires it twice concurrently and asserts the end state and the side-effect count are both exactly one. If it passes, you have a guard. If it fails, you just found a production incident before production did.

One operation, three hard parts

(Developed example — composite scenario.)

Everything above is one operation — order processing that fires a confirmation email — seen through three lenses. Pulling the fragments into a single readable flow shows how the three techniques compose rather than compete. Assume place_order enqueues the email and returns; a worker sends it; confirm_order advances the state machine.

class FakeEmailer:
    def __init__(self):
        self.sent = []
    def send(self, to, subject, body):
        self.sent.append({"to": to, "subject": subject, "body": body})
    def last_sent(self):
        return self.sent[-1] if self.sent else None


def test_order_processing_all_three_hard_parts():
    emailer = FakeEmailer()                                   # external dep: a fake we control

    order = place_order(cart, customer, emailer=emailer)

    # 1) ASYNC — wait for the condition, not a fixed delay
    wait_until(lambda: emailer.last_sent() is not None)
    assert emailer.last_sent()["to"] == customer.email

    # 2) EXTERNAL DEP — assert against the recording, deterministically
    assert len(emailer.sent) == 1
    assert "confirmation" in emailer.last_sent()["subject"].lower()

    # 3) STATE — provoke the race, assert the invariant holds
    with ThreadPoolExecutor(max_workers=2) as pool:
        [f.result() for f in [pool.submit(confirm_order, order.id) for _ in range(2)]]
    assert get_order(order.id).status == "confirmed"
    assert len(emailer.sent) == 1                             # still one — no second email

Read the three failure modes this one test would have caught in the original sleep(2) version. The async assertion fails if the email fires late — a real bug the sleep was hiding. The external-dependency assertions fail if the wrong address or a duplicate goes out — impossible to check reliably against a live mail server. The state assertions fail if two confirmations each send an email — the double-charge class that ships when nobody tests concurrency. None of these are flakiness. All three were always bugs; the old test couldn't see them because it depended on the clock instead of the conditions.

The pattern behind the three

Line the hard parts up and the same shape appears every time: something varies between runs, and the technique removes the variance by making the test control the input instead of observing it.

The hard part	Why it resists unit testing	The technique that cracks it
Async	The result isn't ready when the assertion runs; timing varies with machine load	Wait for the condition, not a duration — await the result, or poll the post-condition with a bounded timeout
External dependency	Slow, flaky, and non-deterministic; a failure tests their uptime, not your code	Use the right double — a fake that records what it was asked to do, verified against reality by one integration test
State	One-call-at-a-time tests never exercise the interleaving that corrupts data	Provoke the transition — fire concurrent operations and assert one legal end state, once

The column that ties it together is the middle one. Each hard part is hard for the same underlying reason: a real-world input the test failed to control leaked in — the clock, the network, or the order. That's why the discovery question at the top works on all three. This is the same mechanism the End-to-End Testing Without the Pain article describes at the browser level, where a sleep waiting for a page render loses the same race against the same uncontrolled clock. The level changes; the diagnosis and the fix don't.

Takeaway you can run today: for your next flaky test, write the varying input in the test name itself — test_x_when_email_is_slow, test_x_under_concurrent_confirm. Naming the variance forces you to control it, and a test that controls its inputs can't be flaky.

What to do next

You don't need to refactor the whole suite this week. Do this in order, and stop when the noise stops:

1. Grep for sleep(. Replace each one with a wait_until on the real post-condition. This alone kills most timing flake and speeds the suite up.

2. Find your most-retried test. Ask the discovery question: clock, network, or order? Apply the matching row from the table.

3. Pick one at-most-once transition and race it. If there's no guard, you've found a latent double-charge; add the conditional update and keep the test.

4. Delete one @retry. Retries hide exactly the bugs this article teaches you to catch. Remove the retry, watch what fails, and fix the dependency instead of masking it.

The prerequisites for all of this — test isolation, one assertion of intent per test, injecting dependencies rather than reaching for them — are in Unit Testing Best Practices: A Practical Guide. Get those right and the hard parts get a lot less hard.

A flaky test is not a fact of life or a tax you pay for testing async code. It's a message: this test depends on something it doesn't control. Name the clock, the network, or the order — then take it away.

Sources

The order-processing scenario, the CI failure, and the code in this article are a composite — an illustrative example built from patterns common to async, dependency-heavy, and stateful test suites. It does not describe a specific company or incident. The techniques (wait-for-condition over fixed delay, test doubles for external services, and concurrency tests that assert invariants) are standard practice reflected in the documentation of major test frameworks such as Playwright, pytest, and JUnit; consult your framework's guidance for the exact API in your stack.

Keep learning. This article is part of the Software Testing Foundations path in the ShiftQuality Learning Center.