Post-Implementation Reviews That Actually Improve Things

A post-implementation review produces a document. Whether it produces an improvement is a separate question, and there is exactly one number that answers it.

Aldermoor Health ran a PIR after every significant change for two years — proper ones, with a facilitator, an agenda, and action items that had owners and dates.

Then someone read eighteen of them end to end. "The deployment runbook was out of date" appeared in seven.

Each time it had been written up as a finding. Each time an action item said update the runbook, with a named owner and a due date. Each time it was completed and ticked off. And it came back, on average, every eleven weeks.

Nothing about the process had failed. The meetings happened, the findings were real, the actions were done. What had failed was quieter: every one of those action items fixed the instance and left the mechanism alone.

The repeat finding is the only PIR metric worth keeping

Teams try to measure PIRs by attendance, by action items closed, by whether the meeting happened at all. All three go up while nothing improves — Aldermoor scored perfectly on every one of them for two years.

The measurement that works is recurrence: for each finding, how many previous reviews said something equivalent?

• First occurrence — new information. The review did its job.

• Second — possibly coincidence, possibly a pattern. Worth noting.

• Third or more — the review is documenting a problem it is not solving, and has been for months.

A recurring finding is not a nag. It is a specific, diagnosable signal: the action item was scoped to the event rather than to the thing that produces the event. "Update the runbook" fixes today's runbook. Runbooks go stale continuously, so the fix has to be continuous too — Aldermoor ended up failing the deploy when the runbook's last-verified date exceeded 90 days, and that finding has not returned since.

The test you can run: open your last six reviews and put every finding in one list. Anything appearing three times is your most valuable open problem, and the fact that it keeps being closed is the evidence that it isn't.

The recurrence audit

An afternoon's work, and it usually reframes an entire year of reviews.

Finding	Reviews it appeared in	The action item written	What it addressed
Runbook out of date	7 of 18	"Update the runbook"	the instance — runbooks decay continuously
Nobody told support	5 of 18	"Email support before release"	the instance — depended on memory
Rollback took too long	4 of 18	"Document the rollback steps"	the instance — never rehearsed
Migration ran long	2 of 18	"Add a buffer to the estimate"	plausibly the instance; watch it
Config drift between envs	1 of 18	"Fix staging config"	genuinely one-off

Read the right-hand column. Every recurring row has an action that would have to be repeated forever to work, performed by someone who has to remember. Every one of them has a mechanism-level counterpart that runs on its own: a staleness check in the pipeline, support on the release checklist as a blocking item, a rollback rehearsed as part of the change rather than described in a document.

The two rows at the bottom are what a healthy list looks like — findings that appear once or twice and then stop.

The test: for each recurring finding, ask what would have to be true for it never to recur. If the answer requires a person to remember something, it will recur.

What makes an action item survive the meeting

Most reviews produce action items. Most action items produce nothing, and the failure modes are consistent enough to check against directly.

• Vague. "Improve communication." Nobody can verify it, so nobody can fail it. If you can't say what would be different next Tuesday, it isn't an action.

• Unowned. "The team should…" — which person? A shared owner is no owner.

• Untracked. Captured in the review document and nowhere else. It now depends on somebody rereading the document, which nobody does.

• Aspirational. "We learned we should test more." Was that a new insight, or something everyone already knew and didn't do? The second one isn't a finding; it's a mood.

• Scoped to the instance. The one that produces the seven-out-of-eighteen pattern above.

The four mechanical fixes are dull and they work: one named person, a description specific enough to verify, a due date, and the item lives in the team's normal work system — the same backlog as everything else. An action item that only exists in a review document is a note, and notes lose to sprint work every time.

Three to five items is the working range. Fewer suggests the review didn't surface enough; twenty-five means nothing was prioritised and none will be done.

The test: take your last review's action items and search your issue tracker for each one. Any that aren't there were never going to happen, regardless of how carefully they were written.

The agenda, compressed

Sixty to ninety minutes, and the order matters more than the length — outcomes get assessed before anyone is allowed to explain.

1. Restate the original goal (5 min) — in the words used at the time, not as remembered. The team has been heads-down and the framing has drifted.

2. Walk the timeline (15 min) — what happened, in order, with times. Deploy at T+0, first alert at T+47, noticed at T+52, rolled back at T+1:18. No "we should have" yet.

3. Assess against the success criteria (15 min) — did it do what it set out to do? This is where teams fudge hardest. If the goal was a 30% latency reduction and you got 8%, that is a partial outcome, not a success, and calling it one costs you the finding.

4. What worked (10 min) — specific and reusable. "The pre-flight checklist caught two issues" is a pattern to reproduce. "Everyone did great" is not.

5. What didn't (15 min) — name the thing, not the person.

6. Action items (15 min) — against the four criteria above, checked against the recurrence list.

7. Capture and circulate (5 min) — one or two pages. Long documents don't get read, and a review nobody reads is the recurrence problem in embryo.

The one addition worth making to a standard agenda: before writing any action item, check whether this finding has appeared before. That single question is what turns a review series into a feedback loop rather than a filing system.

The test: in your next review, have last quarter's findings visible in the room. Watch whether anything on the list is about to be written down again.

Where a PIR is the wrong instrument

Reaching for a review in these situations produces a document that satisfies a process and teaches nobody anything.

• A live incident. That's an incident post-mortem — different timing, different attendees, different question. A post-mortem asks why the system failed; a PIR asks whether a deliberate change achieved its purpose. Running one as the other loses both.

• Too soon. Hold it once the outcome is observable. For a latency change that might be days; for an organisational change, weeks. A review held the morning after measures relief, not results.

• A change nobody can state a goal for. If you can't recover what it was supposed to achieve, you have no criterion to assess against, and the review will drift into general impressions.

• A trivial change. Reviewing everything trains people to attend without engaging, which costs you the reviews that matter.

The one case people wrongly skip is the successful change. Successes are where you learn what to reproduce, and the recurrence list gets much of its value from findings on the good side. Skipping them means you only ever learn from damage.

The test: count what fraction of your reviews follow changes that went well. If it's near zero, the process is functioning as an inquest, and people will treat it as one.

What to change before your next review

Don't redesign the meeting. Do the audit.

Put every finding from your last six reviews in one list and count repeats. Take whatever appears three or more times and ask the mechanism question: what would have to be true for this never to happen again? Then write that as the action item instead of the one you wrote last time.

Aldermoor's list had three findings above the line. All three had action items that required a person to remember something on a specific day, and all three now run without anyone thinking about them. Their reviews are shorter, and the documents are no longer the point.

Setting the criteria a review will later be assessed against is done long before the review — that's writing a change request. And where the finding is that people never heard about the change at all, the fix is upstream in the communication plan.