Quality Metrics That Actually Matter

Most engineering orgs have a quality dashboard, and most of them are lying by accident. Test count, up and to the right. Coverage holding at 84%. Forty-seven bugs closed this sprint. Every number is green, every number is real, and not one of them answers the only question leadership actually asked: is the software getting better or worse?

The dashboard can't answer it because it was never built to. It measures how hard the team is working — how many tests they wrote, how many tickets they closed, how many points they burned down. Effort is easy to count. Quality is hard to count. So teams count effort, put it on a wall, and quietly start believing it is quality. That substitution is the single most expensive mistake in how software teams measure themselves, and it survives precisely because everything on the board looks responsible.

This piece is about the substitution — why it happens, the one test that exposes it, and the four metrics that survive that test. If your dashboard is green and your incident channel is busy, you already know the two don't agree. Here's how to build a board where they do.

The trap: measuring activity and calling it quality

Start with the mechanism, because the trap is not stupidity — it's a reasonable chain of decisions that ends in the wrong place.

Leadership wants quality to be visible, so it asks for metrics. The metrics that are easy to instrument are the ones that count actions: tests written, coverage percentage, bugs closed, story points, sprint-commitment hit rate, review approval rate. Each gets a target. And the moment a number becomes a target, people optimize the number — not the thing the number was supposed to stand for.

This is Goodhart's law, sharpened by the anthropologist Marilyn Strathern into the line everyone quotes: "When a measure becomes a target, it ceases to be a good measure." It is not a cynical claim about lazy engineers. It is a structural claim about proxies. A proxy metric and the real goal move together right up until you reward the proxy — then they come apart, because gaming the proxy is almost always cheaper than improving the real thing.

Watch how cheap:

• Tests written rewards volume. A hundred assertions that can't fail is a hundred tests.

• Code coverage measures which lines ran during the suite, not whether any test would notice if that line broke. You can execute a line and assert nothing about it.

• Bugs closed rewards having bugs to close. The team that ships fewer defects has a worse number.

• Sprint-commitment hit rate rewards under-committing. Promise less, hit 100%, look reliable.

• Review approval rate rewards rubber-stamping. "LGTM" at scale.

Every one of these is a real number that goes up when the team performs the ritual, whether or not the software improved. That's the whole disease in one sentence.

Run this this week: take your team's top three dashboard metrics and, for each, write down the laziest way to move it without shipping better software. If you can name that move in under a minute — and you always can — that metric is coaching your team toward the move. Watch which numbers spike right before a review cycle; that's the game already being played.

The one test that sorts activity from outcome

Here is the discovery move — the single question that separates a metric worth trusting from a metric worth deleting:

Does this number go up when people work harder, or when the software gets better?

That's it. Run every metric through it.

Tests written goes up with effort — type more tests, number climbs, software unchanged. Coverage goes up with effort — exercise more lines, number climbs, nothing caught. Bugs closed goes up with effort and with having more bugs, which is worse than useless. All three fail: they respond to labor, not to outcome.

Now the flip side. A metric passes the test only if the cheapest way to improve it is to actually improve the software. If gaming it is harder than fixing the real thing, the incentive finally points where you want it. That's the property you're hunting for — not "hard to game" in some absolute sense, but "gaming it costs more than doing the work honestly."

This question is also a discovery instrument, in the sense this blog keeps returning to: most measurement failures are a scoping failure that happened before anyone wrote a query. Nobody asked what outcome are we actually trying to see before they picked what to count. They reached for what the tools emitted by default. Ask the effort-versus-outcome question at the start — when you're deciding what goes on the board — and you never build the lying dashboard in the first place. Ask it after, and you're doing remediation. (This is the same intake-first discipline behind what we mean by "shift quality": the metric you skipped defining is the failure you'll debug later.)

What that looks like when it goes wrong

(Developed example — composite scenario.)

A payments team I'll call the composite for a mid-market SaaS company had a board they were proud of, and they'd earned the pride. Coverage sat at 84% and had for a year — they gated merges on it. They closed an average of 47 bugs a sprint, tracked on a burndown that always burned down. Velocity was stable. In sprint reviews, the quality slide was three green numbers and a nod, and the room moved on.

Then a director who'd been burned before asked for one number nobody had on the wall: of every defect the team fixed last quarter, what fraction was first reported by a customer in production, versus caught internally before release? It took an afternoon to reconstruct from the issue tracker. The answer, quarter over quarter, was 22%, then 29%, then 38%. The escaped-defect rate had been climbing the entire time the dashboard stayed green.

Nothing on the board had moved because nothing on the board was watching the outcome. Coverage was flat at 84% — but the 84% was made of tests that ran the payment code and asserted almost nothing about it. Bugs closed was high — because more bugs were escaping to production, getting reported by customers, and coming back as tickets to close. The activity metrics weren't just missing the decline. They were counting the symptom of the decline as success. Every escaped defect became a customer ticket, and every closed customer ticket made "bugs closed" look better.

What they changed was small and specific. They stopped gating on coverage percentage and started gating on whether new code came with tests that fail when you break the code — they ran a mutation-testing check on changed files, exactly the honest-coverage move detailed in Test Coverage: The Honest Version. Coverage as a headline number came off the wall. Escaped-defect rate went up, as a tracked trend line, not a target with a threshold. Within two quarters the trend bent back toward the low twenties — not because anyone was told to hit a number, but because the number they now watched only moved when the software genuinely caught more before release. The dashboard finally agreed with the incident channel.

The lesson isn't "escaped-defect rate good." It's that the outcome metric revealed, in one afternoon, what a year of green activity metrics had hidden — and it did so because you cannot fake it without actually shipping fewer defects.

The four metrics that survive the test

Four metrics pass the effort-versus-outcome question. One is a ratio you can compute from your own issue tracker; three are the delivery metrics from Google's DORA research program, validated across tens of thousands of respondents in the Accelerate / State of DevOps work. None of them rewards typing. Each ends with something to compute this week.

Read every one of these as a trend, not a scoreboard. The absolute value depends on your domain, your risk tolerance, and how you define an incident — a payments platform and an internal dashboard tool will never share a target. What's comparable is your own line over time. Is it bending the right way? That question is honest across every context. A threshold ("keep it under X") re-creates the Goodhart trap you just escaped.

1. Escaped defect rate

Of all the defects your team found in a period, what fraction reached production before anyone caught them? Production defects divided by total defects found. It is the most honest quality signal most teams don't track, and it passes the test cleanly: the only way to improve it is to catch more defects before release, which is the software getting better.

Compute this week: pull last quarter's resolved defects, tag each as "found before release" or "found in production," and take the ratio. Then do the prior quarter and the one before. You don't want a number — you want three dots and a direction.

2. Change failure rate

What percentage of your deployments cause a problem — an incident, a rollback, a hotfix, a degraded service? This is a DORA metric, and it resists gaming because the only cheap way to lower it is to ship changes that break less often. You can't type your way out of a failed deploy.

Compute this week: count your deploys over the last month and how many required remediation. Divide. Track it monthly. If it climbs while your activity board stays green, the activity board is the one that's wrong.

3. Time to restore service (MTTR)

When something does break, how long until service is restored? Another DORA metric. It measures your recovery machinery — detection, diagnosis, rollback, communication — none of which improves by writing more tests or closing more tickets. It improves by building real operational capability.

Compute this week: for your last five incidents, record detection-to-recovery time. Watch whether the trend shortens as you invest in observability and rollback tooling. Five data points and a slope beat one impressive average.

4. Lead time for changes

How long from a commit landing to that change running in production? The last of the four DORA metrics. It exposes the health of your whole delivery pipeline — review latency, build reliability, deploy friction. Unlike velocity, which rewards estimating generously, lead time is wall-clock reality and can only be shortened by removing genuine friction.

Compute this week: sample ten recent changes, measure commit-to-production for each, take the median. Re-sample monthly. The trend tells you whether your pipeline is getting faster or quietly silting up.

The swap, side by side

The move in every case is the same: replace the metric that rewards effort with the metric that rewards outcome. Same intent, honest instrument.

Gameable activity metric	Why it's gameable	Outcome metric that replaces it
Tests written	Volume of assertions, none of which need to be able to fail	Escaped defect rate — did tests actually catch defects before production?
Code coverage %	Measures which lines ran, not whether a test would notice a regression	Escaped defect rate (paired with mutation testing on changed code)
Bugs closed	Rewards having bugs; more escapes means more tickets to close	Change failure rate — did the change break production?
Story points / velocity	Rewards generous estimation and under-committing	Lead time for changes — wall-clock commit to production

Keep the activity numbers if you find them useful for capacity planning — they're fine as diagnostics. The rule is narrower than "activity metrics are bad": never put an effort metric on the wall as a stand-in for quality, and never attach a target to one. The target is what weaponizes Goodhart's law.

How to use these without creating new number-fakers

The failure mode of this entire piece is obvious and worth naming: pick escaped-defect rate, slap a target on it — "under 15% or the sprint fails" — and you've rebuilt the trap one layer up. Teams will reclassify production defects as "enhancements," argue about what counts as an escape, and stop logging the awkward ones. Goodhart doesn't care which metric you chose; it cares that you turned a measure into a target.

Three guardrails keep the outcome metrics honest:

• Trend, never threshold. Report the direction of the line and talk about why it's moving. The moment a number becomes a pass/fail gate for a team's reputation, it becomes something to manage rather than something to learn from.

• Own it above the team. These are system metrics. Escaped-defect rate and change failure rate reflect the whole delivery pipeline — architecture, process, staffing, tooling. Pin them to an individual or a single squad and you've just taught people to hide defects instead of surfacing them. This is where quality gates that actually gate earn their place: gate on evidence the change is safe, not on a person's number.

• Pair the metric with the mechanism. Escaped-defect rate tells you that defects are getting through; it doesn't tell you where. Read it alongside a real testing strategy — where each class of bug should have been caught — like the layered approach in testing strategies that scale. And when you weigh the cost of chasing a metric down, weigh it honestly, the way the cost of quality, honestly frames it — some escapes are cheaper to absorb than to prevent.

The point of a quality metric is not to have a number. It's to know, on any given week, whether the software is getting better or worse — and to find out before your customers do. If your current board can't answer that, you don't need more metrics. You need to run the four you have through one question, and put the survivors on the wall as trends.

Sources

• Goodhart's law, in the form popularized by Marilyn Strathern (1997), "'Improving Ratings': audit in the British University system," European Review 5(3): "When a measure becomes a target, it ceases to be a good measure." Originally attributed to economist Charles Goodhart (1975).

• DORA metrics (change failure rate, time to restore service, lead time for changes, deployment frequency): Google Cloud's DevOps Research and Assessment program and Accelerate: The Science of Lean Software and DevOps (Forsgren, Humble, Kim, 2018), plus the annual State of DevOps / DORA reports.

• The payments-team scenario is a developed composite built from patterns common across delivery teams — not a single real company, and its figures (84% coverage, 47 bugs/sprint, the 22%→29%→38% escaped-defect trend) are illustrative, not measured.

Keep learning. This article is part of the Quality Management Fundamentals path in the ShiftQuality Learning Center.