Designing a Test Strategy for a Growing Team

The build is red again. A payments engineer scans the failure — a checkout test that timed out waiting for a modal — shrugs, and clicks Re-run jobs. Eleven minutes later it's green. Nobody changed anything, and nobody knows why it failed or why it passed the second time. This is a normal Tuesday on a team that grew from five engineers to twenty without anyone deciding how testing should work.

The suite takes twenty-two minutes and fails roughly one run in four for reasons unrelated to the code under review. So the team has learned a reflex: red, re-run; green, merge. The safety net has quietly become a slot machine — nobody trusts a red build enough to stop for it. That is the exact moment a real bug ships behind a "flaky" label.

A test strategy is what a team adopts before that Tuesday becomes every day: not more tests, but a decision about where confidence comes from and who owns it as the team scales. This piece is about that organizational transition — why a suite that worked at five breaks at twenty, and the moves that keep a shared suite fast and trusted. It is deliberately not a tour of the test layers (Testing Strategies That Scale lays out that taxonomy) or a document template (Test Strategy Document: A Working Template) — read those for the what. This is about when it stops working, and what to do.

Why "no strategy" is a strategy at five and a liability at twenty

At five engineers, the absence of a written test strategy is not a gap. Everyone has touched every file. Review catches design problems because the reviewer already holds the whole system in their head. When something breaks, the person who broke it usually wrote it and can fix it that afternoon, because a failure points at one obvious place.

None of those conditions survive the trip to twenty. The system is bigger than any single head. The reviewer of a payments change may never have opened the notifications service it calls. The engineer who breaks a test at 4 p.m. may not be the person who wrote it or able to tell in thirty seconds whether the failure is real. The properties that made "no strategy" work were properties of small headcount, not of good engineering — and they decay silently, so the team keeps operating on habits that no longer hold.

What changes as the team grows	5 engineers	20 engineers
System knowledge	Everyone holds the whole system	No one holds the whole system; knowledge is siloed by service
Who diagnoses a failure	The author, who wrote it yesterday	Often a stranger to that code, mid-review, under time pressure
Suite trust	A red build means a real problem	A red build means "probably flake — re-run it"
Release fear	Low; blast radius is small and understood	High; nobody is sure what a change touches, so every release feels risky
Cost of a wrong test	One person's afternoon	A shared tax paid by everyone who runs the suite, forever

The last row does the damage. At five engineers a badly-placed test costs you time; at twenty it costs everyone time, on every run, indefinitely — and no single person feels enough of it to fix it. That diffusion of cost is the mechanism behind everything that follows.

Try this: ask any engineer to explain, out loud, what your three slowest tests actually verify. If they can't, your suite has already outgrown the shared knowledge that used to hold it together.

The doom loop, and the economics underneath it

When a suite starts failing for reasons unrelated to the change, the intuitive response is test more. It feels responsible. It is exactly wrong, for a mechanical reason. Every test has a cost — to write, to run on every commit, to maintain when the code around it moves — and a value — the bugs it catches and the confidence it buys. A unit test runs in milliseconds, fails for one clear reason, and runs a thousand times a day for free. An end-to-end test costs seconds to minutes, fails for a dozen unrelated reasons (slow network, shifted selector, leftover data), and is expensive to diagnose. Same nominal purpose, wildly different economics.

The doom loop is what happens when a team adds value without accounting for cost:

1. A bug escapes to production.

2. Someone writes a test where the bug was visible — usually the top, through the UI, because that's where it was reported.

3. The suite gets slower and flakier by one more brittle test.

4. Lower speed and higher flake mean lower trust; lower trust means people re-run instead of investigate.

5. Re-running instead of investigating means the next escaped bug also gets a top-level test.

6. Return to step 1, now slower and less trusted than before.

Each decision is defensible. The aggregate is a twenty-two-minute suite that catches less than the five-minute one did, because nobody reads its failures anymore. The loop runs on treating "we added a test" as the goal instead of "we added confidence per unit of cost."

Try this: pull your ten slowest tests and, for each, ask what unique bug it's the only thing catching. Any test that's slow, flaky, and duplicated by a cheaper test below it is pure cost — a candidate to delete this week, and deleting it makes the suite faster and more trusted at once.

One bug's journey up the pyramid

(Developed example — composite scenario, drawn from patterns common to teams making this transition.)

A team of twenty ships a bug: on checkout, orders with a discount code applied are charged the full, pre-discount amount. Support tickets arrive, and a money bug gets attention fast.

The reflex fix. An engineer reproduces it in the browser and — because that's where they saw it — writes an end-to-end test: spin up a browser, register a user, add an item, apply code SAVE10, check out, assert the total is $45 and not $50. It's a real test; it would have caught this. Green, ship, move on.

That test now costs roughly 40 seconds on every run and needs a browser, a seeded product, a live discount record, and a payment sandbox that occasionally times out. Over the next few months, three more money bugs escape and get the same treatment — four 40-second browser tests, each flaky in its own way, now at the top of the suite. This is the pyramid inverting in real time, because escaped bugs are always most visible at the top.

The same bug, caught at the right level. Where did the discount actually break? Not in the browser, the checkout page, or even the API — in a single pricing function computing the payable total. The fault was one line: total = subtotal where it should have read total = subtotal - discount. A unit test — apply_discount(subtotal=50, code="SAVE10") returns 45 — runs in about 2 milliseconds, needs no browser, database, or sandbox, and fails for exactly one reason with the broken line named in the stack trace.

Here's the before/after the reflex hides:

	Reflex E2E test	Right-level unit test
Run time	~40 s	~2 ms
Fails because of	timeouts, selectors, data, network, or the bug	the bug
Tells you	"checkout is wrong, somewhere"	"apply_discount returns the wrong total"
Cost across 4 similar bugs	~160 s added to every run, four flake sources	~8 ms, four precise signals

The unit test is not a worse version of the E2E test. It catches the same class of bug faster, more precisely, and without adding a flake source — while the reflex path buys identical protection at twenty-thousand times the runtime and hands you a vague failure instead of a line number. Multiply that across twenty engineers each "just adding one E2E test" after each incident, and you have manufactured the twenty-two-minute suite from the opening scene, one reasonable decision at a time.

Try this: take your last production bug and locate the single function where the fault actually lived — not where it was reported. If a test at that level would have caught it, that's where the test belongs, regardless of where the symptom appeared.

The discovery move: "what's the lowest level that could have caught this?"

That question is the whole strategy compressed into one habit, worth naming as a deliberate move rather than a moral. Every time a bug escapes, before anyone writes a test, the team asks: what is the lowest, cheapest level that could have caught this? The test goes there, not where the bug happened to surface.

This is a discovery question, not a testing question. Bugs are reported at the top (a user saw a wrong number) but live at the bottom (a function returned one). The reflex writes the test where the bug was reported; the discipline writes it where the bug lived. The discount bug was reported at checkout and lived in a pricing function — and the only way you get to the two-millisecond unit test is by asking the discovery question before reaching for the browser.

There's a diagnostic hiding in the answers. If it keeps coming back "only end-to-end could have caught this," the pieces underneath aren't independently verifiable — a design signal: logic tangled into the view, or service contracts that no contract test pins down. The escaped-bug review surfaces those design problems as a side effect.

Try this: add one line to your incident template — "Lowest level that could have caught this: ___" — and make filling it in a precondition for closing the ticket. Do this for a month and the pattern of where your bugs live tells you where your suite is thin.

Ownership has to move, or the strategy stays on paper

At five engineers, ownership is implicit and it works: you broke it, you own it. That fails at twenty, because the person who hits a red build is frequently not the author of the failing test, and they face a bad choice — spend an hour understanding someone else's test, or re-run and merge. Under deadline pressure, they re-run. Diffuse ownership is what powers the doom loop; "just re-run it" is a rational response to a suite that belongs to no one.

Whatever the model — service-team ownership, a rotating build cop, a shared flaky-test budget — the strategy has to answer four questions explicitly, because at twenty nobody answers them by default:

• Who fixes a failing shared test — the author, the last committer, the owning team, or a rotating cop?

• Who can delete a test that's more cost than value — and are they allowed to, without a meeting?

• Who watches suite health — run time and flake rate — as a number someone owns, not a vibe?

• What happens when a test is flaky — a written policy, or does everyone just re-run?

A test no one owns is a test no one will fix, and an unfixed flaky test is a slow leak in the whole suite's credibility. This is the human-factors half of the problem — the half a document about test layers can't solve.

Try this: name, today, the one person or rotation responsible for suite health — run time and flake rate — with standing authority to delete a test that costs more than it's worth. If you can't name them, that's your first strategy decision, and it's free.

Keeping a shared suite fast and trusted

Speed and trust are the same problem viewed twice. A fast suite gets run; a slow one gets skipped. A suite whose red builds mean something gets investigated; a flaky one gets re-run. Both roads end at a safety net nobody uses, so both deserve a named threshold rather than a shrug. Three keep a shared suite honest as headcount grows:

• A run-time budget with a number. Set the ceiling — say, the fast suite stays under ten minutes per PR — and when it's breached, move tests down to cheaper levels or out to a nightly run rather than accept the new normal. The layering that makes this possible is in Testing Strategies That Scale; the point here is that the budget is a decision someone enforces.

• A flaky-test policy with teeth. A test that fails intermittently for non-code reasons is worse than no test — it trains the team to ignore red. The policy: a test that flakes past a set rate is quarantined out of the blocking suite within a day and fixed or deleted, on a clock, by the owner named above. What you must not do is paper over it with automatic retries — auto-retrying until green hides the exact race conditions that become production incidents (End-to-End Testing Without the Pain walks through that retry trap).

• A split between what blocks and what informs. Fast, reliable tests block PRs; slow or third-party-dependent tests run on merge or nightly and inform. This keeps the blocking path fast enough to stay trusted without discarding the slow tests' coverage.

Try this: put two numbers on a dashboard this week — median run time and flake rate (runs that pass only on re-run) — and read them at every retro. A climbing flake rate is a suite losing trust, whether or not anyone has said so out loud.

What to automate, and what to leave to people

Automation is a cost as much as a value, so the same economics apply. Automate a check when it runs many times, has a clear pass/fail, and protects something that would hurt to break: regression, critical-path flows, contract tests, and — especially — a test for every bug that has already escaped once, placed at the lowest level that would have caught it. Leave to people what automation is bad at: exploratory testing and any judgment about whether a result is good rather than merely unchanged. Automating an exploratory session doesn't make it cheaper; it makes it a brittle script that tests one frozen path and calls it coverage.

Try this: before automating any new check, write one sentence naming the specific bug it stops from recurring. If you can't, it's automation for its own sake — cost the growing suite can't afford — and it belongs in a person's exploratory pass instead.

Where this doesn't apply

The pyramid-and-economics default is not universal law, and a strategy that pretends otherwise gets quietly ignored. Three genuine exceptions:

• Very small teams and early products. At five engineers on an unsettled product, formal strategy is overhead; shared knowledge and review do the job. The signal to adopt one is growth plus pain — a suite people have started to distrust — not headcount alone. If the model choice itself is unsettled, read Test Pyramid vs. Test Trophy first.

• Integration-heavy systems. For a service that is mostly glue between other services, the highest-value tests cluster at the integration and contract level — a "test trophy," not a pyramid. The economics hold; the cheapest level that catches the bug is simply higher up.

• Safety- or money-critical paths. Some flows justify redundant coverage at multiple levels on purpose, because the blast radius warrants it. The discipline isn't "never duplicate" — it's "duplicate deliberately, knowing what it costs," not by reflex after every incident.

Try this: before adopting the default, write one sentence on which of these three your system is. The answer changes the shape of the target — where the cheapest catch lives — even though the cost-versus-value rule doesn't move.

What to do Monday

You don't need a strategy document to start — you need to interrupt the doom loop once and show the team it can be interrupted. In order:

1. Put two numbers on a wall: median suite run time, and flake rate (runs that pass only on re-run). You can't manage the loop you can't see.

2. Name an owner for suite health, with authority to delete a test that costs more than it's worth — no meeting required.

3. Add the discovery line to your incident template — "Lowest level that could have caught this: ___" — as a precondition for closing a ticket.

4. Run the escaped-bug audit once: take the last five production bugs, find where each fault actually lived, and move or add a test at that level, deleting the reflex top-level test a cheaper one now covers.

5. Only then write it down, using the Test Strategy Document template, so the next ten hires inherit the decision instead of re-learning it on a red Tuesday.

A test strategy for a growing team isn't a bigger suite. It's the small set of decisions — where confidence comes from, who owns it, and what "fast enough" and "trusted enough" mean as numbers — that keep twenty engineers sharing one suite without turning it into a slot machine nobody believes.

Sources

This article uses a developed composite example — the discount-pricing bug and its journey up the pyramid — built from patterns common to teams scaling from roughly five to twenty engineers. It is illustrative, not a case study of any specific organization, and the runtime figures (~40 s for a browser test, ~2 ms for a unit test) are order-of-magnitude illustrations of the cost gap between test levels, not measured benchmarks. The economic framing is standard test-strategy reasoning, developed further in testing strategies that scale and choosing a pyramid or trophy model.

Keep learning. This article is part of the Software Testing Foundations path in the ShiftQuality Learning Center.