Set Up Coverage Reporting — Test Automation in Practice, Part 5

Test Automation in Practice · Part 5

Coverage is the most gamed number in testing: 90% can mean thoroughly tested or thoroughly meaningless, depending on whether those executed lines actually assert anything. Used well, though, a coverage report is a map of the code your tests never touch. This walks through setting up coverage reporting, wiring it into CI, and reading it for the real gaps — without sliding into worshiping the percentage.

Coverage tells you what code your tests executed. Not what they verified — those are different. This tutorial wires it up and walks through reading it correctly.

What You'll Build

Coverage tracking integrated into your test run, with reports visible in CI.

Step 1: Install Coverage Tooling (5 min)

Python:

pip install pytest-cov

Node:

vitest has coverage built in via --coverage.

npm install --save-dev @vitest/coverage-v8

Step 2: Configure (5 min)

Python — add to pytest.ini:

[pytest]
addopts = --cov=yourapp --cov-report=term-missing --cov-report=html

Vitest — in vitest.config.ts:

export default defineConfig({
  test: {
    coverage: {
      provider: 'v8',
      reporter: ['text', 'html', 'lcov'],
      include: ['src/**/*'],
      exclude: ['src/**/*.test.*', 'src/**/*.spec.*'],
    },
  },
});

Step 3: Run and Read (10 min)

pytest

or

npm test -- --coverage

Output includes per-file coverage percentages and total.

Open htmlcov/index.html (or coverage/index.html). Browse:

• Files sorted by coverage

• Click into a file to see line-by-line

• Red lines: not exercised

• Yellow: partially exercised (branches)

Step 4: Find the Real Gaps (15 min)

Click into low-coverage files. For each red line, ask:

• Is this code that runs in production? → Should be tested.

• Is this error handling for unreachable cases? → Document; consider removing.

• Is this defensive code for impossible inputs? → Consider removing.

Don't add tests for every red line. Add tests for red lines that matter.

Step 5: Set Coverage Thresholds (10 min)

For critical code, fail builds below a threshold:

Python in pytest.ini:

[pytest]
addopts = --cov=yourapp --cov-fail-under=80

Vitest:

coverage: {
  thresholds: {
    statements: 80,
    branches: 75,
    functions: 80,
    lines: 80,
  },
}

Set thresholds at the level appropriate for your code. Critical modules higher; experimental lower.

Step 6: Add to CI (10 min)

In your GitHub Actions workflow:

- name: Run tests with coverage
  run: pytest --cov=yourapp --cov-report=xml

- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v4
  with:
    file: ./coverage.xml

Codecov (or alternative) shows coverage trends, per-PR diffs, and posts comments on PRs.

Step 7: Configure Per-File Targets (10 min)

Different code deserves different coverage:

# codecov.yml
coverage:
  status:
    project:
      default:
        target: 80%
    patch:
      default:
        target: 90%  # New code coverage

component_management:
  individual_components:
    - component_id: critical
      paths:
        - src/auth/**
        - src/payments/**
      statuses:
        - type: project
          target: 95%

Critical paths get higher bars.

Step 8: Avoid the Coverage Worship Trap (5 min)

Things to remember:

• 100% coverage with weak assertions tests nothing

• 80% with strong assertions tests a lot

• Coverage on unimportant code doesn't matter

• Coverage on critical code matters a lot

Don't write tests just to hit numbers. Write tests for behaviors that matter.

Step 9: Branch Coverage > Line Coverage (5 min)

If your tool offers both, focus on branch coverage. A line with if x: can be 100% line-covered with one test (just enters the if). Branch coverage requires both true and false paths.

For most projects, configure branch coverage as the meaningful metric.

Step 10: Periodic Coverage Audit (ongoing)

Quarterly:

• Walk the lowest-coverage files

• Decide: needs coverage, or shouldn't be in the codebase?

• Walk the highest-coverage files

• Check: meaningful tests, or coverage-padding tests?

The combination keeps the suite honest.

What You Just Did

You have coverage tracking that surfaces gaps without becoming a target you game. The reports are diagnostic; the team responds to them deliberately.

Common Failure Modes

Coverage as target. "Must hit 90%." Tests written just to satisfy. Quality drops.

Vanity coverage. Tests that import a file and assert nothing. Coverage rises; tests verify nothing.

Ignoring branch coverage. Line-covered, branch-uncovered code. Bugs hide.

Coverage thresholds set arbitrarily. "We picked 80%." Match to risk: critical code higher; experimental lower.

Coverage reports nobody reads. Configured but never opened. Build it into the team's PR review.

---

Continue the Test Automation in Practice path

• Previous — Part 4: Mock External Dependencies Cleanly

• Next — Part 6: Run Tests in Parallel

Part of the Test Automation in Practice learning path.