Designing a Test Strategy for a Growing Team

Contributor
Mar 15
5 min read

At five engineers, you don't need a test strategy. Everyone knows the whole codebase, code review catches most things, and if something breaks, the person who wrote it fixes it before lunch. Testing is whatever each person decides it is, and that's fine.

At twenty engineers, that same non-strategy is actively hurting you. No one knows the whole system anymore. The person who broke something isn't the person who has to debug it. The test suite — accreted from twenty people's private habits — is slow, flaky, and trusted by no one. Releases get scary. The team's instinct is to test more, which makes the suite slower, which makes people trust it less. That's the doom loop a real test strategy exists to break.

This post is about designing one as you scale: not a 40-page QA manual, but a set of decisions about what to test where, what to automate, and how to keep the whole thing fast enough that people actually use it.

The strategy is mostly about economics

Every test has a cost (to write, to run, to maintain) and a value (the bugs it catches, the confidence it gives). A test strategy is, underneath, a set of decisions about getting the most confidence per unit of cost. That framing cuts through most testing arguments.

A unit test runs in milliseconds, fails for one clear reason, and costs almost nothing to run a thousand times a day. An end-to-end test runs in seconds or minutes, can fail for a dozen unrelated reasons, and costs real time and flakiness every run. Neither is "better" — they buy different things. The strategy is knowing which to reach for, and the default rule is simple: push every check to the cheapest level that can actually catch the bug.

The pyramid, and why teams invert it

The testing pyramid — lots of unit tests, fewer integration tests, few end-to-end tests — is old advice, and it's still right, because the economics haven't changed. The failure mode is that growing teams invert it without meaning to.

Here's how it happens. A bug escapes. Someone writes an end-to-end test to catch it, because that's the level the bug was visible at. Repeat a hundred times across twenty engineers, and you've built an upside-down pyramid: a thick layer of slow, flaky end-to-end tests sitting on a thin base of unit tests. The suite now takes twenty minutes, fails randomly, and everyone has learned to re-run it until it's green — which means it's catching nothing.

The discipline is to resist the easy reflex. When a bug escapes, ask: what's the lowest level that could have caught this? Usually it's a unit or integration test, not an end-to-end one. Catch it there. Reserve end-to-end tests for a small number of genuinely critical user journeys — the flows where "all the pieces work together" is the actual thing you need to verify.

What to automate, and what not to

Automation is not the goal. Confidence is the goal, and automation is how you buy confidence for the repetitive parts so humans are free for the parts only humans can do.

Automate anything you'll run more than a few times with a clear pass/fail: regression checks, critical-path flows, contract tests between services, and — especially — anything tied to a past incident. A bug that happened once will happen again; a test that pins it down is one of the highest-value tests you can write.

Keep human the things humans are uniquely good at: exploratory testing (poking at a feature to find what the spec didn't anticipate), usability judgment, and the "does this feel right?" check that no assertion captures. A team that automates these away ships software that passes every test and still feels broken.

The line moves as you grow. Early on, manual testing of the critical path is fine. By twenty engineers, the critical path changes too often for anyone to re-test by hand, so it has to be automated — not because automation is virtuous, but because manual no longer scales.

Protect the suite's speed and trust

Two properties make or break a test suite at scale: it has to be fast, and it has to be trusted. Lose either and people route around it, at which point it's just a tax.

Speed is a feature you have to defend on purpose. A suite that takes thirty minutes won't be run locally, which means failures get discovered in CI, which means slower feedback and more context-switching. Parallelize, push tests down the pyramid, and treat a slow suite as a bug to be fixed, not a fact of life.

Trust is destroyed by flaky tests faster than by anything else. A test that fails 5% of the time for no real reason teaches the entire team to ignore red — and once "red" means "probably nothing," your suite has stopped working as a safety net even when it's right. Adopt a hard rule: a flaky test gets fixed or deleted, immediately. A deleted flaky test is better than a flaky one, because at least it doesn't lie.

Make "tested enough" an explicit decision

The most useful thing a test strategy gives a growing team is a shared answer to "is this tested enough to ship?" Without one, every change reopens the argument, and the answer depends on who's reviewing.

Write it down. It can be as simple as: new logic gets unit tests; anything touching the critical path gets an integration test; the checkout and auth flows have end-to-end coverage; every fixed bug gets a regression test. That's a policy a twenty-person team can apply consistently without a meeting. (The Test Strategy Template in this path gives you a structure for capturing exactly these decisions — levels, automation boundaries, ownership, and your definition of "tested enough" — on a page or two. Pair it with the Test Case Tracker to keep per-feature coverage visible.)

The point of writing it down isn't bureaucracy. It's that a decision made once, in calm, beats the same decision re-litigated under deadline pressure by whoever happens to be in the review.

Let the strategy grow with the team

A test strategy is not a one-time artifact. The right strategy at five engineers is "use judgment." At twenty it's "here's our pyramid, our automation boundary, and our definition of done." At fifty it's that plus ownership boundaries, contract testing between teams, and a flakiness budget. Revisit it whenever the team roughly doubles, because the constraints that shaped the last version have changed.

The teams that scale testing well aren't the ones with the most tests. They're the ones who decided, on purpose, what to test where — and kept the suite fast and trustworthy enough that people actually believed it. Get those decisions written down and defended, and your test suite becomes what it's supposed to be: the thing that lets you move fast because you're not afraid of what you'll break.

ShiftQuality