Tutorial 5: Set Up Coverage Reporting

Contributor
3 days ago
3 min read

Coverage tells you what code your tests executed. Not what they verified — those are different. This tutorial wires it up and walks through reading it correctly.

What You'll Build

Coverage tracking integrated into your test run, with reports visible in CI.

Step 1: Install Coverage Tooling (5 min)

Python:

pip install pytest-cov

Node:

vitest has coverage built in via --coverage.

npm install --save-dev @vitest/coverage-v8

Step 2: Configure (5 min)

Python — add to pytest.ini:

[pytest]
addopts = --cov=yourapp --cov-report=term-missing --cov-report=html

Vitest — in vitest.config.ts:

export default defineConfig({
  test: {
    coverage: {
      provider: 'v8',
      reporter: ['text', 'html', 'lcov'],
      include: ['src/**/*'],
      exclude: ['src/**/*.test.*', 'src/**/*.spec.*'],
    },
  },
});

Step 3: Run and Read (10 min)

pytest

npm test -- --coverage

Output includes per-file coverage percentages and total.

Open htmlcov/index.html (or coverage/index.html). Browse:

Files sorted by coverage
Click into a file to see line-by-line
Red lines: not exercised
Yellow: partially exercised (branches)

Step 4: Find the Real Gaps (15 min)

Click into low-coverage files. For each red line, ask:

Is this code that runs in production? → Should be tested.
Is this error handling for unreachable cases? → Document; consider removing.
Is this defensive code for impossible inputs? → Consider removing.

Don't add tests for every red line. Add tests for red lines that matter.

Step 5: Set Coverage Thresholds (10 min)

For critical code, fail builds below a threshold:

Python in pytest.ini:

[pytest]
addopts = --cov=yourapp --cov-fail-under=80

Vitest:

coverage: {
  thresholds: {
    statements: 80,
    branches: 75,
    functions: 80,
    lines: 80,
  },
}

Set thresholds at the level appropriate for your code. Critical modules higher; experimental lower.

Step 6: Add to CI (10 min)

In your GitHub Actions workflow:

- name: Run tests with coverage
  run: pytest --cov=yourapp --cov-report=xml

- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v4
  with:
    file: ./coverage.xml

Codecov (or alternative) shows coverage trends, per-PR diffs, and posts comments on PRs.

Step 7: Configure Per-File Targets (10 min)

Different code deserves different coverage:

# codecov.yml
coverage:
  status:
    project:
      default:
        target: 80%
    patch:
      default:
        target: 90%  # New code coverage

component_management:
  individual_components:
    - component_id: critical
      paths:
        - src/auth/**
        - src/payments/**
      statuses:
        - type: project
          target: 95%

Critical paths get higher bars.

Step 8: Avoid the Coverage Worship Trap (5 min)

Things to remember:

100% coverage with weak assertions tests nothing
80% with strong assertions tests a lot
Coverage on unimportant code doesn't matter
Coverage on critical code matters a lot

Don't write tests just to hit numbers. Write tests for behaviors that matter.

Step 9: Branch Coverage > Line Coverage (5 min)

If your tool offers both, focus on branch coverage. A line with if x: can be 100% line-covered with one test (just enters the if). Branch coverage requires both true and false paths.

For most projects, configure branch coverage as the meaningful metric.

Step 10: Periodic Coverage Audit (ongoing)

Quarterly:

Walk the lowest-coverage files
Decide: needs coverage, or shouldn't be in the codebase?
Walk the highest-coverage files
Check: meaningful tests, or coverage-padding tests?

The combination keeps the suite honest.

What You Just Did

You have coverage tracking that surfaces gaps without becoming a target you game. The reports are diagnostic; the team responds to them deliberately.

Common Failure Modes

Coverage as target. "Must hit 90%." Tests written just to satisfy. Quality drops.

Vanity coverage. Tests that import a file and assert nothing. Coverage rises; tests verify nothing.

Ignoring branch coverage. Line-covered, branch-uncovered code. Bugs hide.

Coverage thresholds set arbitrarily. "We picked 80%." Match to risk: critical code higher; experimental lower.

Coverage reports nobody reads. Configured but never opened. Build it into the team's PR review.

ShiftQuality