top of page

Improve a Metric — Engineering Metrics & DORA, Part 9

  • Shawn West
  • Jul 28
  • 4 min read

Updated: Aug 20

Engineering Metrics & DORA · Part 9

Measuring a metric is the easy part; moving it is where most teams stall — usually by bolting on process the moment a number looks bad. Improving a metric is an experiment, not a mandate. This walks through doing it deliberately: pick one metric, hypothesize why it's where it is, run one change, measure whether it actually helped — and reverse it if it didn't.

Tracking metrics isn't enough. Pick one; identify the bottleneck; ship the change; verify.

Before you start

  • One metric with at least a quarter of clean history. You cannot detect a change against a baseline you do not trust, and normal variation is wider than most teams expect.

  • The segment-level data from Part 2, or the failure-type breakdown from Part 4. Improving a headline number without knowing which component drives it is guesswork with a dashboard attached.

  • Agreement that you will change one thing at a time. Two simultaneous interventions produce a result you cannot attribute, which means you learn nothing even when it works.

  • A named owner and a review date, both written down before you start. The most common failure of this loop is that nobody comes back to check.

Step 1: Pick One Metric (10 min)

Don't try to improve everything at once.

Pick the one:

  • Most visible to the team

  • Has clear improvement levers

  • Tied to a real pain

Common picks:

  • Lead time (PRs taking too long)

  • MTTR (oncall is brutal)

  • Deploy frequency (deploys feel scary)

Pick one. Focus.

Step 2: Set a Target (10 min)

Current: lead time p50 is 5 days.

Target: lead time p50 is 2 days.

By when: end of Q3.

Specific. Measurable. Bounded.

Stretch but realistic. 5-day → 1-hour overnight = unrealistic.

Step 3: Hypothesize the Cause (15 min)

Why is lead time 5 days?

Hypotheses:

  • Review takes 2 days on average

  • PRs are too big (1 day to review)

  • CI is slow (30 min per push)

  • Deploys are weekly only

For lead time specifically: time-segmented analysis helps (Part 2).

Pick the biggest segment.

Step 4: Pick the Intervention (15 min)

For long review:

  • Smaller PRs

  • Reviewer SLA

  • Async-first conventions

  • Pre-review (author self-reviews)

For slow CI:

  • Parallelize

  • Cache deps

  • Test pyramid (fewer slow tests)

Multiple possible interventions. Pick the one with highest ROI.

Step 5: Communicate the Change (10 min)

Team announcement:

"We're going to reduce lead time from 5d to 2d over Q3.

Specifically:
- PRs under 300 lines (target; flag bigger)
- Reviewer SLA: first response within 4h business hours
- Pre-PR self-review checklist

Trial 4 weeks; reassess."

Get buy-in. Without it: lip service.

Step 6: Try the Change (varies)

Run for the trial period.

  • Encourage adoption

  • Coach individuals

  • Adjust as needed

Don't force-launch a thing nobody understands. Coach.

Step 7: Measure Impact (10 min)

After trial:

  • Did lead time drop?

  • By how much?

  • Any side effects (e.g., quality dropped)?

If the metric improved + nothing else got worse: keep the change.

If it didn't improve: re-hypothesize.

Step 8: Don't Add New Process for Every Issue (10 min)

The instinct: add a checklist; add a meeting; add a rule.

Often: simpler is better.

"Reviewer SLA: first response within 4h."

vs.

"Daily review queue meeting. Triage. Assign. Track in JIRA."

The first might work. The second adds process forever.

Step 9: Reverse Bad Interventions (10 min)

If your change:

  • Didn't help

  • Made something worse

  • Annoyed the team

Reverse. Don't keep dead process.

This requires trust: "we tried; didn't work; rolling back."

Better than fossilized broken practices.

Step 10: Move On to the Next (5 min)

Done with this metric? Repeat:

  • Pick another

  • Pattern: investigate → intervene → measure

A team that compounds these over quarters reaches elite levels.

A team that stares at the dashboard and does nothing: stuck.

The loop, and where each stage fails

Stage

What you actually do

How it fails

Baseline

Record the current value and its normal variation over the last quarter

Only the point value is recorded, so ordinary week-to-week movement later gets read as impact

Hypothesis

Name the specific component you believe dominates, from segment data

The hypothesis is the metric itself — "lead time is too high" is a restatement, not a cause

Intervention

Change one thing, sized so its effect would exceed normal variation

Several changes ship at once and nothing is attributable

Communication

Tell the team what is changing and why, before it changes

Skipped, so the team optimises against a target they were not told about — which is how gaming starts

Measurement

Wait a defined period, then compare against the baseline and its spread

Measured too early, or against a mental impression of "before"

Keep or revert

Reverting is a valid outcome and should be as easy as keeping

Process added for a one-off problem stays forever, and the next team inherits it as tradition

You're done when

  • You can state the baseline and its normal variation, so you know what size of change would count as real rather than as noise.

  • Your hypothesis names a component and a mechanism — not the metric restated as a complaint — and the people affected heard about the change before it landed.

  • After the review period you can say keep or revert with a number behind it, and if you reverted, the process you added is actually gone.

What You Just Did

Improving a metric: pick one, set target, hypothesize cause, intervene, communicate, try, measure, simple > process-heavy, reverse bad ideas, move to next. The improvement loop.

Common Failure Modes

Improve everything at once. Nothing improves.

No target. No way to know if successful.

Intervene without hypothesis. Random changes.

Don't measure impact. Belief vs. data.

Process bloat. Each "improvement" leaves a meeting behind.

Sources

Continue the Engineering Metrics & DORA path

Part of the Engineering Metrics & DORA learning path.

bottom of page