top of page

Instrument MTTR — Engineering Metrics & DORA, Part 5

  • Shawn West
  • Jul 28
  • 5 min read

Updated: Aug 19

Engineering Metrics & DORA · Part 5

Failures are inevitable; what separates strong teams isn't never breaking — it's how fast they recover. Mean time to restore measures exactly that, and it's often a better health signal than trying to prevent every possible incident. This walks through instrumenting MTTR from your incident data, benchmarking it, and — crucially — avoiding the trap of gaming it by quietly closing incidents early.

MTTR is mean time to restore service after an incident. Speed of recovery matters more than incident count.

Before you start

  • Incident records with real timestamps — detection, mitigation, all-clear — for at least a quarter. Fewer than about twenty incidents and you are reading noise, because this distribution has a long tail and the tail is the whole story.

  • A deploy log you can join incidents against. Without it you cannot separate a change-caused impairment from a data-centre problem, and that separation is now what the metric is for.

  • Agreement, in writing, on what "restored" means before you compute anything. Step 1 exists because two engineers will otherwise answer it differently and neither will mention it.

  • Somewhere to plot a distribution. A mean alone will mislead you here, and Step 9 is where that becomes obvious.

Step 1: Define Restore (10 min)

When is the incident "over"?

  • Service back to normal SLO

  • Customer-facing functionality restored

  • Internal monitoring green again

Pick the one that maps to user impact. Document.

Often: a combination — service stable + monitoring green for N minutes.

Step 2: Define Start (10 min)

When did the incident begin?

  • First customer impact (hardest to measure)

  • First alert fired (most common)

  • First detection (varies)

"First alert fired" is automatable. Pick this unless you have better signal.

Step 3: Connect to Incident Management (15 min)

PagerDuty / FireHydrant / Opsgenie:

  • Start = page time

  • End = mark resolved (with sanity check)

API access for data.

incidents = pagerduty.list_incidents(since=last_month)
for inc in incidents:
    duration = inc.resolved_at - inc.created_at
    record_mttr(inc.id, duration)

Step 4: Compute (10 min)

Incident 1: 22 minutes
Incident 2: 45 minutes
Incident 3: 12 minutes
...

Mean: 25 minutes
Median: 22 minutes
p95: 50 minutes

Track median and tail. Mean can be misleading (one 4-hour incident skews).

Step 5: DORA Bands (5 min)

  • Elite: less than 1 hour

  • High: less than 1 day

  • Medium: less than 1 week

  • Low: more than 1 week

Where your team lands depends entirely on which report year you benchmark against, so quote the year with the band. The challenge: consistency.

Step 6: What Improves MTTR (15 min)

  • Better alerting (catch issues earlier)

  • Runbooks (faster response)

  • Practiced oncall (people know the system)

  • Easy rollbacks (one-click)

  • Feature flags (kill switches)

  • Good observability (find the cause)

  • Trained team (calmly handle stress)

MTTR is downstream of these practices.

Step 7: Per-Service / Per-Severity (10 min)

Aggregate hides reality:

Per-service:
  checkout: avg 15min
  search: avg 60min
  admin: avg 4h

admin is slow. Either appropriate (low stakes) or fixable.

Per-severity:

SEV1: avg 30min
SEV2: avg 2h
SEV3: avg 1d

Different urgency; different expected restore time.

Step 8: The Lazy Reset Trap (10 min)

"Mark resolved" doesn't always mean restored:

  • Engineer marked resolved when alert cleared

  • But customer-facing issue persisted

False MTTRs are bad data.

Mitigation:

  • Validate "resolved" against user-facing metrics

  • 15-min auto-reopen if metrics still bad

Step 9: Plot Distribution (10 min)

A histogram tells the story:

  • Most incidents: 5-30 min

  • A few: 1-4 hours (tail)

Aim to shrink the tail. The 1% of incidents that take 4+ hours dominate the user pain.

Step 10: Post-Incident Improvements (5 min)

Each incident: what would have shortened MTTR?

  • Better alerts?

  • Runbook missing?

  • Tools needed?

  • Knowledge in heads of others not on call?

Aggregate findings; act. MTTR improves through pattern.

Where the clock starts and stops

Boundary

The tempting definition

The one that survives a post-mortem

Start

When someone opened the incident

When the impairment began affecting users — often well before anyone noticed. Using ticket-open time silently rewards slow detection

Stop

When the incident ticket was closed

When users were served correctly again. Closing paperwork on Monday for a Friday fix inflates every number in the set

Scope

Any outage the team responded to

Impairment caused by a change you deployed. An upstream provider failing is real, and it is not this metric

Statistic

The mean

The median and the tail together. One four-hour incident drags a mean far enough to hide a month of fast recoveries

The name changed, and so did the definition

This tutorial instruments what DORA originally called mean time to restore. In 2023 DORA renamed it failed deployment recovery time and narrowed it — the older framing did not distinguish a failure your change caused from one caused by something external, such as a data-centre outage, which made it inconsistent with the other delivery metrics.

That is not a cosmetic change, and it is why Step 3 tells you to join incidents to deploys. Scope the clock to change-caused impairment and the number you produce here is the current metric, whatever your dashboard labels it. Leave external outages in and you are measuring your cloud provider alongside your team, then reporting the total as though it were yours.

One consequence for reporting up: if you present this to anyone who has read a recent report, use the current name and say what you scoped it to. "MTTR: 25 minutes" invites a question you should answer before it is asked.

You're done when

  • Every incident in your window has a start and a restore timestamp derived from user impact, not from ticket administration — and you can name the incident where those two differ most.

  • You can say what share of your incidents were caused by a deploy versus something external, because the join in Step 3 actually ran. If you cannot split them, the number is not this metric yet.

  • You have the median and the tail in front of you, not just the mean, and you can point at the small number of long incidents that account for most of the user-facing pain.

What You Just Did

MTTR: define restore + start, connect to incidents, compute, bands, what improves it, per-service / severity, lazy reset, distribution, improvement loop. Recovery speed.

Common Failure Modes

Lazy "resolved" tagging. False MTTR.

Mean only; ignore tail. Long incidents hidden.

Aggregate hides per-service issues. Slow services masked.

Compare across companies blindly. Context matters.

Track but don't learn. Same incidents recur with same MTTR.

Sources

Continue the Engineering Metrics & DORA path

Part of the Engineering Metrics & DORA learning path.

bottom of page