Instrument Change Failure Rate — Engineering Metrics & DORA, Part 4

Engineering Metrics & DORA · Part 4

Shipping fast only helps if you're not shipping breakage — and change failure rate is the metric that keeps the other DORA numbers honest. It's the percentage of deploys that cause a problem: a rollback, a hotfix, an incident. This walks through instrumenting it without the two classic distortions — over-counting every minor blip, and quietly under-counting the failures nobody bothered to tag.

What % of deploys cause incidents? The stability companion to deploy frequency.

Before you start

• A deploy log covering enough releases to be worth dividing. A rate computed over a dozen deploys moves several points every time one fails; over a few hundred it means something.

• Incident and rollback records you can join to specific deploys, by version or timestamp. Without that join every number below is a guess wearing a decimal point.

• A written definition of "failure", agreed before you count anything. Step 1 is the whole tutorial — the two teams who disagree about whether a config fix counts will produce rates that differ by a factor of two from identical pipelines.

• Hot-fix and rollback markers in your process, or the willingness to add them. Untagged, they are indistinguishable from ordinary deploys, which is exactly the under-count Step 7 is about.

Step 1: Define Failure (15 min)

Variations:

• Rollback within 24h

• Hot-fix deploy required

• Any incident attributed to a deploy

• SLO breach traced to a change

Pick one. Document.

Common practice: "rollback within 24h of deploy" — concrete, automatable.

Strict: "any user-impacting incident" — requires incident review.

Step 2: Tag Rollbacks (10 min)

In your deploy automation:

deploy --service=checkout --version=v2.5.1
deploy --service=checkout --version=v2.5.0 --rollback   # marked

Mark explicitly:

• Reason

• Original deploy ID

• Cause (if known)

Tag for analysis.

Step 3: Tag Hot-Fixes (10 min)

deploy --service=checkout --version=v2.5.2 --hotfix-for=v2.5.1

A deploy specifically to fix the previous one. Counts as failure of v2.5.1.

Without tagging: indistinguishable from normal deploys.

Step 4: Connect Incidents to Deploys (15 min)

Incident management:

Incident #5678
- Started: 14:33 UTC
- Service: checkout
- Last deploy before: v2.5.1 at 14:31
- Confirmed cause: yes; deploy

Connection lets you compute: per-deploy failure rate.

Most incident tools (PagerDuty, FireHydrant) support tagging.

Step 5: Compute the Rate (10 min)

Failed deploys: 5
Total deploys: 200
Failure rate: 2.5%

Per service. Per team. Over a window (28 days typical).

Bands:

• Elite: 0-15%

• High: 16-30%

• Medium / Low: 16-30%

Step 6: Beware Over-Counting (10 min)

Don't count as failures:

• Planned rollbacks (canary didn't work; planned)

• Test in production that didn't promote

• Routine cleanup

Do count:

• Rollback after errors increased

• Hot-fix for the previous deploy

• Customer-impacting incidents

Strict definition; consistent.

Step 7: Beware Under-Counting (10 min)

Don't miss:

• Silent failures (didn't roll back; suffered through)

• "Partial outages" the team brushed off

• Deploys followed by "config changes" (sometimes mask rollback)

If failure rate is suspiciously low: dig deeper.

Step 8: Failure Type Categories (10 min)

After 6 months: review categories:

• Database migration issues: 30%

• Frontend regression: 25%

• Config / env vars: 20%

• 3rd-party integration: 15%

• Other: 10%

Patterns. Now you know where to invest:

• Better migration tooling?

• Stronger E2E tests?

Step 9: Failure Rate by Service (10 min)

checkout: 5%
search: 12%
payments: 2%
admin: 25%

admin is bad. Or maybe expected (it's experimental).

Bench against your service's risk tolerance.

Step 10: Visualize and Improve (5 min)

Dashboard:

• Failure rate trend (4-week rolling)

• Per-service breakdown

• Failure causes (last quarter)

• Top services for investment

Visible. Discussable. Improvable.

The two distortions, side by side

The same pipeline produces wildly different rates depending on which of these you fall into — which is why comparing your number to another team's is close to meaningless unless both definitions are on the table.

	Over-counting	Under-counting
What gets miscounted	Planned rollbacks, canary aborts that worked as designed, and follow-up deploys for a single incident counted as separate failures	Silent fixes rolled into the next feature deploy, config changes made outside the pipeline, and failures nobody linked back to the release that caused them
Why it happens	Any deviation from the happy path is treated as breakage, including the safety mechanisms doing their job	Fixing quietly is faster than filing, and the tagging discipline in Steps 2 and 3 was never set up
What it looks like on a dashboard	A rate that spikes whenever the team ships something genuinely risky and handles it well	A flat, respectable rate that nobody trusts and that never moves
How it distorts behaviour	Teams stop using canaries and staged rollouts, because using them looks like failing	The metric stops detecting anything, and the first signal is an incident review

Sizing a fix without inventing a number

Once the failure-type breakdown in Step 8 names your biggest category, the obvious next question is what a fix would buy you. Resist answering it with a percentage. Nobody can know in advance what better migration tooling or a stronger end-to-end suite will recover, and a made-up figure becomes the number your sponsor holds you to.

Do this instead. Take the failures in your largest category and ask, one at a time, whether the proposed change would have caught this specific one. Twelve migration failures, eight of which a schema-diff check in CI would have blocked, gives you a defensible sentence: eight of our last twelve migration failures were preventable by one check. That is an audit of real incidents, not a forecast, and the reader can go verify it. Then ship the change and watch the rate — the intervention loop in Part 9 exists precisely so the impact is measured after the fact rather than promised before it.

Reading the bands without misreading them

When you benchmark, one thing catches people out and gets reported as a typo: in DORA's published tables the high and medium clusters have shared a change failure rate band. That is a real finding rather than an error — change failure rate is the metric that separates the top clusters least cleanly, so it is the weakest of the four to rank teams by.

Two practical consequences. Band thresholds have changed between report years, so name the year beside any band you quote and check the current report before quoting at all. And if you need to show improvement on this metric, show the trend of your own rate against your own definition, not your position in a tier.

You're done when

• Your written definition of failure survives three real edge cases from your own history — a planned rollback, a config change made outside the pipeline, and an incident fixed in the next feature deploy — and you can say how each is counted.

• Rollbacks and hot-fixes are tagged at deploy time, so the rate is computed from markers rather than reconstructed from memory.

• You can name your largest failure category and say how many of its recent instances a specific proposed change would have prevented — a count from your own incidents, not an estimated percentage.

What You Just Did

Change failure rate: define failure, tag rollbacks and hot-fixes, connect incidents, compute, avoid over/under-counting, categorize, per-service, visualize. Stability signal.

Common Failure Modes

Vague definition. Inconsistent count.

No automation. Manual tagging missed.

Just one number org-wide. Hides per-service issues.

Don't categorize. Don't know where to fix.

Track and not act. Numbers on a wall.

---

Sources

• DORA, DORA's software delivery performance metrics — the current definitions of change fail rate and the wider metric set.

• DORA, Research — the State of DevOps report archive. The performance bands sit inside individual annual reports and differ by year; quote the year with the band.

Continue the Engineering Metrics & DORA path

• Previous — Part 3: Instrument Deploy Frequency

• Next — Part 5: Instrument MTTR

Part of the Engineering Metrics & DORA learning path.