Maintain a Long-Lived Test Suite — Test Automation in Practice, Part 10

Test Automation in Practice · Part 10

Every test suite starts as an asset and, left alone, drifts into a liability — slow, flaky, and welded to implementation details nobody remembers choosing. The fix isn't heroics; it's a light, scheduled maintenance habit. This walks through a quarterly audit that keeps a long-lived suite fast, trustworthy, and actually catching bugs — instead of just running and being tolerated.

A test suite that worked great in year 1 can become a millstone by year 3. Without maintenance, suites accumulate stale tests, slow paths, and false confidence. This tutorial walks through preventing decay.

What You'll Build

A maintenance practice: quarterly audits, deletion discipline, and continuous improvement.

Step 1: Establish a Quarterly Audit (15 min)

A 2-hour exercise every quarter. Walk the suite asking:

• Which tests have been failing intermittently?

• Which tests reference removed features?

• Which tests duplicate other tests?

• Which tests no one understands?

• Which tests are testing the wrong thing?

Schedule it now. Recurring calendar event.

Step 2: Delete Aggressively (varies)

The hardest discipline: deleting tests.

Candidates for deletion:

• Tests for code that's been removed

• Tests that always pass but exercise nothing meaningful

• Tests that fail on every release but only ever for "expected" reasons

• Tests with no clear owner who left the company

• Tests that nobody understands and nobody is willing to investigate

Each deletion makes the suite faster, more trustworthy, and easier to maintain.

The fear: "what if it was catching something?" If you can't articulate what, it probably isn't.

Step 3: Identify Slow Tests (15 min)

Profile your suite:

pytest --durations=20

Shows the 20 slowest tests. Look at them.

• Why is each one slow?

• Could it be faster?

• Is the slowness inherent or accidental?

Often, 10% of tests consume 80% of suite time. Optimizing the slow ones pays off.

Step 4: Look for Implementation-Coupled Tests (20 min)

Walk recently-changed tests. For each:

• Why did this test change?

• Was it a behavior change or an implementation refactor?

If most test changes track refactors, the suite is implementation-coupled. Schedule investment in decoupling.

Step 5: Update Documentation (10 min)

The test strategy doc from Part 1. Still accurate?

• Coverage allocations still right?

• Tools still current?

• Roles still match?

• Out-of-scope still relevant?

Update what's stale.

Step 6: Review Flakiness Trends (10 min)

From your quarantine tracking:

• How many flakes detected this quarter?

• Trending up or down?

• Common patterns?

• Outstanding quarantined tests?

Address patterns; clear backlog.

Step 7: Verify Tests Still Catch Bugs (15 min)

Look at recent production bugs:

• Could a test have caught this earlier?

• Were tests written for the bug after the fact?

• Are there patterns of bugs slipping through?

If many bugs escape testing, the suite has gaps. Diagnose and fill.

Step 8: Refresh the Test Strategy (20 min)

Once per year (not quarterly):

• Does the test pyramid shape still fit?

• Are the right things being tested?

• Has the team's context changed (size, product, tools)?

• What would we do differently if starting over?

Adjust the strategy. Communicate changes.

Step 9: Train New Team Members (per onboarding)

When someone joins:

• Walk through the test strategy doc

• Show how to run tests at each level

• Demonstrate the quarantine process

• Explain the philosophy (behavior over implementation, etc.)

Tribal knowledge dies. Documented and onboarded knowledge persists.

Step 10: Celebrate Maintenance (ongoing)

The work of maintaining is invisible. Most people only notice the suite when it's broken.

Make maintenance visible:

• Mention test improvements in team meetings

• Credit engineers who delete or refactor tests

• Track the suite health metrics over time and share

A team that values maintenance does it. A team that doesn't, won't.

Maintenance Checklist Template

# Quarterly Test Maintenance — Q[N] [YEAR]

Date: [date]
Facilitator: [name]

## Deletion
- [ ] Removed tests for deprecated features
- [ ] Removed tests that always pass without meaningful assertions
- [ ] Removed duplicate tests

## Performance
- [ ] Profiled suite
- [ ] Optimized top-10 slowest where possible
- [ ] Updated speed targets if needed

## Coupling
- [ ] Audited recently-changed tests
- [ ] Identified implementation-coupled tests
- [ ] Scheduled refactor work

## Flakiness
- [ ] Reviewed quarantine list
- [ ] Cleared resolved entries
- [ ] Identified patterns

## Strategy
- [ ] Strategy doc reviewed
- [ ] Updates applied
- [ ] Communicated changes to team

## Health Metrics
- Suite time: [X] (target: [Y])
- Pass rate: [X]% (target: 99%+)
- Quarantine count: [N] (target: <5)
- Coverage: [X]% (target: per file)

## Action Items
[Tracked in normal work system]

What You Just Did

You set up the practice that keeps test suites useful for years. Maintenance isn't glamorous; it's what separates teams whose tests they trust from teams whose tests they tolerate.

Common Failure Modes

No maintenance. Suite rots invisibly. Year 3 it's a millstone.

Maintenance without deletion. Tests accumulate; suite slows; nobody removes anything.

Maintenance without metrics. Without measurement, you don't know if you're improving.

Owner-less suites. Nobody specifically owns suite health. Falls through cracks.

Strategy ignored. Updated annually but team doesn't follow. Either enforce or remove.

You're Done

You've moved from a few tests to a comprehensive, maintainable test suite. Strategy, allocation, coverage, parallelization, smoke, visual regression, flake quarantine, and ongoing maintenance.

Recommend Requirements Engineering Workshop next if you want to apply the same hands-on patterns to a different domain.

---

Continue the Test Automation in Practice path

• Previous — Part 6: Run Tests in Parallel

Part of the Test Automation in Practice learning path.