Tutorial 7: Build a Smoke Test Suite

Contributor
Apr 27
4 min read

A smoke test runs fast and verifies the application is fundamentally working. If smoke fails, something basic is broken — no point running deeper tests. This tutorial walks through building one.

What You'll Build

A 5-15 test smoke suite that runs in under 5 minutes and gates every deploy.

Step 1: Identify Critical Flows (15 min)

What are the top 3-5 things your application must do?

For a typical SaaS:

User can sign in
Main dashboard loads with user data
Primary action can be completed (e.g., create a workspace, view a report)
Primary integration works (e.g., payment processing in test mode)
Sign out works

These are smoke candidates. Each one is a flow, end-to-end.

Step 2: Pick the Test Level (10 min)

Smoke tests can be at different levels:

API smoke: hits the API directly. Fast, no UI dependency.
UI smoke: clicks through the actual interface. Catches UI breakage.
Mixed: API for setup, UI for the critical visible behaviors.

For most products, API smoke + 1-2 UI E2E checks of the most critical flows.

Step 3: Write the Smoke Tests (30 min)

// tests/smoke/smoke.spec.ts
import { test, expect } from '@playwright/test';

test.describe('@smoke', () => {
  test('application loads', async ({ page }) => {
    const response = await page.goto('/');
    expect(response?.status()).toBe(200);
  });
  
  test('user can sign in', async ({ page }) => {
    await page.goto('/signin');
    await page.fill('[data-testid="email"]', 'smoke@example.com');
    await page.fill('[data-testid="password"]', process.env.SMOKE_PASSWORD);
    await page.click('[data-testid="submit"]');
    await expect(page).toHaveURL(/.*\/dashboard/);
  });
  
  test('dashboard shows user data', async ({ page }) => {
    await signIn(page);
    await page.goto('/dashboard');
    await expect(page.locator('[data-testid="user-name"]')).toBeVisible();
  });
  
  test('primary action works', async ({ page }) => {
    await signIn(page);
    await page.goto('/dashboard');
    await page.click('[data-testid="create-new"]');
    await page.fill('[data-testid="name"]', 'Smoke test workspace');
    await page.click('[data-testid="save"]');
    await expect(page.locator('text=Smoke test workspace')).toBeVisible();
  });
});

API-level smoke:

# tests/smoke/test_api_smoke.py
import pytest
import requests

BASE_URL = os.environ["SMOKE_BASE_URL"]
SMOKE_TOKEN = os.environ["SMOKE_TOKEN"]

@pytest.mark.smoke
def test_health_endpoint():
    response = requests.get(f"{BASE_URL}/health")
    assert response.status_code == 200
    assert response.json()["status"] == "ok"

@pytest.mark.smoke
def test_auth_endpoint():
    response = requests.get(
        f"{BASE_URL}/api/me",
        headers={"Authorization": f"Bearer {SMOKE_TOKEN}"}
    )
    assert response.status_code == 200

@pytest.mark.smoke
def test_database_reachable():
    # Endpoint that hits the DB
    response = requests.get(
        f"{BASE_URL}/api/workspaces",
        headers={"Authorization": f"Bearer {SMOKE_TOKEN}"}
    )
    assert response.status_code == 200

5-15 tests. Each verifies one critical thing.

Step 4: Make It Fast (15 min)

Smoke should be fast. Target: <5 minutes for the whole suite.

Strategies:

Parallel execution (Tutorial 6)
Skip slow setup: use API for setup, UI only for the actual smoke check
Reuse browser session across tests where possible
Minimal data: don't seed large fixtures

If a smoke test exceeds 30 seconds, ask whether it's really smoke or it should be in the regular E2E.

Step 5: Wire to Deploy Pipeline (15 min)

Smoke runs after every deploy. In your CI config:

deploy:
  steps:
    - name: Deploy to staging
      run: ./deploy.sh staging
    
    - name: Wait for service
      run: ./wait-for-healthy.sh
    
    - name: Run smoke tests
      run: pytest tests/smoke -m smoke
      env:
        SMOKE_BASE_URL: https://staging.example.com
        SMOKE_TOKEN: ${{ secrets.SMOKE_TOKEN }}
    
    - name: Rollback if smoke failed
      if: failure()
      run: ./rollback.sh staging

The pattern: deploy → smoke → continue or rollback.

Step 6: Run Continuously in Production (15 min)

Smoke isn't just for deploys. Run it every few minutes against production as synthetic monitoring:

# .github/workflows/synthetic.yml
name: Production smoke

on:
  schedule:
    - cron: '*/5 * * * *'  # every 5 minutes
  workflow_dispatch:

jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest tests/smoke -m smoke
        env:
          SMOKE_BASE_URL: https://example.com
          SMOKE_TOKEN: ${{ secrets.PROD_SMOKE_TOKEN }}
      
      - if: failure()
        run: ./alert-oncall.sh "Production smoke failed"

Or use a dedicated synthetic monitoring service (Pingdom, Datadog, etc.).

Step 7: Make Smoke Test User Visible (5 min)

The smoke test exercises real flows. Make sure the test user is identifiable so support doesn't try to help it:

Use a +smoke email convention
Document the test user in your runbook
Filter out smoke traffic from product analytics

Step 8: Maintain (ongoing)

Smoke tests rot when flows change. Add it to your release checklist:

When a critical flow changes, update the smoke test first
When a smoke test breaks, fix immediately
When you add a new critical flow, add a smoke test

Stale smoke is worse than missing smoke — it passes while real things are broken.

What You Just Did

You built a fast safety net for every deploy. If something fundamental breaks, you know within minutes — before customers do.

Common Failure Modes

Too comprehensive. "Smoke" suite of 100 tests, 30-min runtime. Defeats the purpose. Keep tight.

Too shallow. 1-2 tests that don't actually verify functionality. Misses obvious breakage.

Flaky smoke. Real or perceived flakiness leads to ignored failures. Fix flakes immediately.

No rollback wired. Smoke fails; deploy continues anyway. Failed gates that don't gate.

Smoke user data in production. Test transactions visible in customer-facing reports. Filter.

Next Tutorial

Visual regression catches what functional smoke can't: Tutorial 8: Add Visual Regression Testing.

ShiftQuality