Run Tests in Parallel — Test Automation in Practice, Part 6

Test Automation in Practice · Part 6

A test suite that takes twenty minutes is a suite people quietly stop running. Parallelizing it is the fastest way to win that time back — but done carelessly it surfaces every hidden order-dependency and shared-state bug you didn't know you had. This walks through parallelizing safely: measure first, flush out the isolation problems, and land a suite that's fast and still trustworthy.

A 20-minute test suite that runs in 5 minutes via parallelization is a different experience. Engineers run tests more often. PR feedback is faster. This tutorial walks through enabling parallel execution without breaking the suite.

What You'll Build

Test suite running in parallel, with isolation that prevents tests from interfering with each other.

Step 1: Measure Current Time (5 min)

time pytest

or

time npm test

Note the time. You'll compare after.

Step 2: Enable Parallel Runner (5 min)

Python:

pip install pytest-xdist

Run with auto-parallel:

pytest -n auto

Vitest:

Parallel by default. To control:

vitest --pool=threads --poolOptions.threads.maxThreads=4

Playwright:

// playwright.config.ts
export default defineConfig({
  workers: process.env.CI ? 2 : undefined,
});

Default workers vary by tool. Auto-detection usually works.

Step 3: Run and Measure (5 min)

time pytest -n auto

Typically 2-4x speedup on a multi-core machine. If you're not seeing improvement, you have a bottleneck (likely shared state).

Step 4: Find Order-Dependent Tests (15 min)

Parallel execution exposes order-dependence. Tests that "passed" because of order now fail.

# Run tests in random order
pytest --random-order

# Or specifically with parallel
pytest -n auto --random-order

Failing tests are now visible. Each one is a bug — a test that depended on shared state.

Step 5: Fix the Isolation Issues (varies)

Common causes and fixes:

Shared database state.

Bad: tests use the same DB rows.

Fix: use transactions per test (rolled back), or use unique IDs per test.

@pytest.fixture
def db_session(db_engine):
    connection = db_engine.connect()
    transaction = connection.begin()
    session = Session(bind=connection)
    yield session
    session.close()
    transaction.rollback()
    connection.close()

Shared filesystem.

Bad: tests write to /tmp/test-file.txt.

Fix: use per-test temp directories.

def test_file_writing(tmp_path):
    output = tmp_path / "output.txt"
    write_file(output)
    assert output.read_text() == "..."

Shared in-process state.

Bad: module-level mutable state, singletons.

Fix: reset state in fixtures, or refactor to avoid global state.

Shared external resources.

Bad: tests use the same Stripe test customer.

Fix: unique resources per test, or sequential tests for shared-resource cases.

Step 6: Database Parallelism (30 min)

Postgres-backed integration tests can run in parallel with care:

Option A: per-test transactions (fastest)

@pytest.fixture
def db_session(db_engine):
    connection = db_engine.connect()
    transaction = connection.begin()
    yield Session(bind=connection)
    transaction.rollback()
    connection.close()

Works for most cases. Doesn't work when code-under-test commits its own transactions.

Option B: per-worker schemas (fast)

Each parallel worker gets its own schema:

@pytest.fixture(scope="session")
def db_engine(worker_id):
    schema = f"test_{worker_id}"
    engine = create_engine(f"postgresql://.../{schema}")
    Base.metadata.create_all(engine)
    yield engine
    engine.dispose()

Option C: per-test database (slowest, cleanest)

Spin up a fresh DB per test. Use for tests that really need full isolation.

Step 7: Tune the Worker Count (10 min)

More workers ≠ faster. Past CPU count, overhead exceeds gains.

# Try different counts
pytest -n 2
pytest -n 4
pytest -n 8

Measure each. Pick the sweet spot. Typically cpu_count for CPU-bound; 2 * cpu_count for I/O-bound.

Step 8: Handle Tests That Can't Parallelize (5 min)

Some tests inherently can't run in parallel:

• Tests that exercise shared external resources (e.g., a real third-party with rate limits)

• Tests that mutate global state

• Tests requiring specific resource configurations

Mark them serial:

@pytest.mark.serial
def test_external_resource():
    ...

And run them separately:

pytest -n auto -m "not serial"
pytest -n 1 -m "serial"

Step 9: Verify in CI (10 min)

Update your workflow to use parallel:

- run: pytest -n auto

Check CI time before and after. Should drop significantly.

If CI runners are limited cores (e.g., 2), -n auto may not help much. Consider running test groups in parallel jobs:

jobs:
  test-unit:
    runs-on: ubuntu-latest
    steps:
      - run: pytest tests/unit
  
  test-integration:
    runs-on: ubuntu-latest
    steps:
      - run: pytest tests/integration

Two jobs run in parallel automatically.

Step 10: Watch for Flakiness After (ongoing)

Parallel execution can expose flakiness. Track:

• Tests that pass in serial but fail in parallel

• Tests that fail intermittently in parallel

• Tests that pass everywhere but slow CI dramatically when parallel

Each is information. Fix or quarantine.

What You Just Did

You cut test suite time by 2-4x without sacrificing reliability. Engineers run tests more often. PR feedback is faster. The change pays back daily.

Common Failure Modes

Insufficient isolation. Parallel tests interfere. Diagnose and fix.

Too many workers. Past CPU count, throughput drops. Tune.

Database deadlocks. Concurrent tests deadlock. Use per-worker schemas or transactions.

Resource exhaustion. Tests exhaust connections, file handles. Investigate and tune limits.

Test flake hidden by retries. Auto-retry on failure hides the parallelism-exposed bugs. Don't retry; fix.

---

Continue the Test Automation in Practice path

• Previous — Part 5: Set Up Coverage Reporting

• Next — Part 10: Maintain a Long-Lived Test Suite

Part of the Test Automation in Practice learning path.