top of page

Tutorial 9: Debug a Failing Test

  • Contributor
  • 3 days ago
  • 3 min read

A test failed. The temptation is to look at the error and start guessing. The systematic process is faster.

What You'll Build

A repeatable diagnostic flow for test failures. Each step narrows the problem.

Step 1: Read the Error Message Slowly (1 min)

Most test runners give you:

  • What test failed

  • What assertion failed

  • What was expected vs. what was received

  • A stack trace

Read it carefully before doing anything. Often the answer is right there.

FAILED tests/test_discount.py::test_premium_customer_gets_10_percent_discount

assert discount == 10
E       assert 0 == 10

The discount was 0, not 10. That's a specific clue.

Step 2: Reproduce Locally (2 min)

If the failure was in CI:

pytest tests/test_discount.py::test_premium_customer_gets_10_percent_discount

Just that one test. Quick feedback.

If it passes locally but fails in CI:

  • Environment difference (versions, env vars)

  • Order dependence (other tests pollute state)

  • Timing (CI is slower)

Step 3: Check If It's Order-Dependent (2 min)

Run alone vs. with others:

# Alone
pytest tests/test_discount.py::test_premium_customer_gets_10_percent_discount

# With others
pytest tests/test_discount.py

If alone passes, with-others fails → order dependence. Likely shared state.

Step 4: Add Diagnostic Output (5 min)

For a stubborn failure, add prints:

def test_premium_customer_gets_10_percent_discount():
    price = 100
    tier = "premium"
    
    discount = calculate_discount(price, tier)
    print(f"DEBUG: discount = {discount}, type = {type(discount)}")
    
    assert discount == 10

Run with pytest -s to see prints. Sometimes the print reveals the issue immediately.

For more structured debugging:

import logging
logging.basicConfig(level=logging.DEBUG)

Step 5: Use the Debugger (5 min)

For tests where prints aren't enough:

def test_premium_customer_gets_10_percent_discount():
    price = 100
    tier = "premium"
    
    import pdb; pdb.set_trace()  # breakpoint
    
    discount = calculate_discount(price, tier)
    assert discount == 10

Run the test. At the breakpoint:

  • n step to next line

  • s step into function

  • p discount print value

  • c continue

The debugger lets you inspect state at any point.

For Node/Jest:

test('discount', () => {
  debugger;
  const discount = calculateDiscount(100, 'premium');
  expect(discount).toBe(10);
});

Run with node --inspect-brk node_modules/.bin/jest --runInBand. Open Chrome DevTools.

Step 6: Bisect the Code (10 min)

If the function is complex, narrow it down.

def calculate_discount(price, tier):
    print(f"price={price}, tier={tier}")
    if price <= 0:
        print("returning 0 due to price <= 0")
        return 0
    if tier == "premium":
        result = price * 0.10
        print(f"premium: result={result}")
        return result
    print("returning 0, no tier matched")
    return 0

Run the test. The prints tell you which branch was taken.

If you see "returning 0 due to price <= 0" but price was 100, you've found something interesting. Maybe price is a string "100", not int 100.

Step 7: Check the Test Itself (3 min)

Sometimes the test is wrong. Read it carefully.

  • Are inputs what you think?

  • Is the expectation correct?

  • Are you comparing the right values?

A surprising portion of "bug in code" cases turn out to be "test was wrong."

Step 8: Bisect the History (10 min)

If the test used to pass:

git bisect start
git bisect bad             # current state
git bisect good <commit>   # last known good

Git checks out commits between; you mark each good or bad. Eventually pinpoints the commit that broke it.

Faster than reading every commit.

Step 9: Distinguish "Wrong" from "Flaky" (5 min)

If a test fails inconsistently:

  • Run it 10 times. Does it always fail? Sometimes?

  • Run in different orders.

  • Run in CI vs. local.

Consistent failure: real bug. Investigate.

Inconsistent: flaky. Different problem (covered in Tutorial 10 of Test Automation in Practice). Quarantine; investigate separately.

Step 10: Fix the Underlying Issue (varies)

Once you've identified the cause:

  • Code bug: fix it. Test should pass.

  • Test bug: fix the test. Code is fine.

  • Test data bug: fix fixtures.

  • Environment bug: make environment consistent (containerize, version pin).

  • Order dependence: fix isolation.

After the fix, run the test 5 times to confirm it's stable.

Step 11: Capture What You Learned (5 min)

For non-obvious bugs:

  • Add a comment to the test explaining the edge case

  • Add a similar test for adjacent edge cases

  • Update the function's documentation if behavior changed

  • If the bug pattern could recur, add detection (linter, schema check)

What You Just Did

You replaced guess-and-check debugging with a systematic process. Each step narrows the problem. Most failures yield in 5-15 minutes when worked methodically; the same failures take hours when worked randomly.

Common Failure Modes

Skipping to the debugger immediately. Often the error message has the answer. Read it first.

Adding prints; not removing them. Tests become noisy. Clean up after.

Fixing the symptom. Test was checking for X; X is wrong; "fix" the test to check for Y (whatever the code returns). Now the test documents the bug.

Not reproducing. Try to debug from CI logs alone. Reproduce locally.

Ignoring flakiness. "It'll probably pass next time." It won't, eventually.

bottom of page