Tutorial 5: Set Up CI to Run Tests

Contributor
3 days ago
3 min read

Tests that don't run on every change drift toward broken. CI runs them automatically, catching breakage when it happens. This tutorial walks through setting up CI with GitHub Actions.

What You'll Build

A GitHub Actions workflow that runs your test suite on every PR and on every push to main. PRs show pass/fail status before merge.

Step 1: Verify Tests Run Locally (5 min)

Before going to CI, run tests locally:

pytest           # or
npm test         # or
dotnet test

All should pass. CI runs the same command; if local doesn't work, CI won't either.

Step 2: Create the Workflow File (10 min)

mkdir -p .github/workflows

Create .github/workflows/test.yml:

name: Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: |
          pip install -e ".[dev]"
      
      - name: Run tests
        run: pytest

For Node:

name: Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      
      - run: npm ci
      - run: npm test

Step 3: Commit and Push (5 min)

git add .github/workflows/test.yml
git commit -m "Add CI workflow for tests"
git push

On the next push, GitHub will run the workflow. Check the Actions tab.

Step 4: Verify CI Runs (5 min)

Open your repo on GitHub → Actions tab. You should see the workflow running.

Click through to see:

Each step's output
Time per step
Pass/fail status

The first run takes a few minutes (cache cold). Subsequent runs are faster.

Step 5: Make a PR to Test (10 min)

git checkout -b ci-test
echo "# noop" >> README.md
git add README.md
git commit -m "Test CI"
git push -u origin ci-test

Open a PR. The Actions check appears on the PR. Should pass.

Step 6: Make Tests a Required Check (5 min)

Settings → Branches → Branch protection rules → main:

Require status checks to pass before merging
Add test as a required check
Require branches up to date before merging

Now PRs can't merge if tests fail. The team's discipline is enforced by the tool.

Step 7: Add Caching (10 min)

CI is slow because it reinstalls dependencies every run. Cache them.

For Python (already shown):

- name: Set up Python
  uses: actions/setup-python@v5
  with:
    python-version: '3.11'
    cache: 'pip'

For Node:

- uses: actions/setup-node@v4
  with:
    node-version: '20'
    cache: 'npm'

Cuts CI time by 30-60% for typical projects.

Step 8: Add Coverage Reporting (15 min)

Track how much code is exercised by tests.

For pytest:

- name: Run tests with coverage
  run: pytest --cov=yourapp --cov-report=xml

- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v4
  with:
    file: ./coverage.xml

For vitest:

- run: npm test -- --coverage
- uses: codecov/codecov-action@v4

Coverage is a leading indicator, not a target. Use to find gaps, not to claim victory.

Step 9: Handle Test Database (15 min)

Integration tests need a database. Use service containers:

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: test
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -e ".[dev]"
      - run: pytest
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test

The service starts before steps run.

Step 10: Monitor CI Time (ongoing)

Watch how long CI takes. Targets:

Under 5 minutes for typical projects
Under 10 minutes for larger ones

When CI is slow, developers stop running tests locally — they wait for CI. Vicious cycle. Invest in speed.

What You Just Did

You set up automated testing on every commit. Bugs get caught at PR time, not after merge. The team's discipline is enforced by the tooling.

Common Failure Modes

Tests pass locally, fail in CI. Environment differences. Use containers and pin versions.

CI slow. Cache dependencies; parallelize.

CI optional. Tests can fail and merge still happens. Make tests a required check.

Flaky tests in CI. Different from local; investigate.

Coverage as target. "Must hit 80%" pushes engineers to write low-quality tests for coverage.

ShiftQuality