Post-Implementation Reviews That Actually Improve Things

Contributor
May 22
6 min read

A post-implementation review (PIR) is a structured look-back at a change to ask: did it work, what did we learn, what would we change. Most PIRs are checkbox exercises that produce documents nobody reads. The ones that aren't follow a few specific patterns.

This guide is how to run a PIR that justifies the meeting time.

What a PIR Is For

The PIR exists to answer three questions:

Did the change achieve its purpose? Beyond "did it deploy" — did it move the metric, solve the problem, satisfy the stakeholders.
What did we learn about the system, the team, or the process? Insights that apply beyond this specific change.
What should we change about how we work? Specific, actionable adjustments for the future.

If the meeting doesn't produce answers to these, it didn't justify itself.

When to Run One

Not every change deserves a PIR. The investment should match the change.

Warrants a PIR:

Major migrations or platform changes
Changes that caused unexpected incidents
First-of-its-kind patterns the team might repeat
Changes that affected customer behavior measurably
Multi-month initiatives

Doesn't warrant a PIR:

Routine deploys
Small bug fixes
Standard-pattern changes that went as expected

For changes in between, a lightweight version — a short async write-up — is often appropriate.

Timing the Review

Two competing pressures: review early enough that memory is fresh, late enough that the change's effects are visible.

A working pattern:

Initial review: 1-2 weeks after deployment. Capture the immediate experience while it's fresh.
Follow-up review: 6-12 weeks after. The longer-term effects are now visible.

For incident-driven PIRs (something went wrong during the change), do the initial review within days, while details are still recoverable.

What to Bring to the Meeting

A useful PIR has data before the discussion starts:

The original goal. What were we trying to achieve, in the words of the change request.
The success criteria. What we said would prove it worked.
The measured outcomes. What actually happened, on the metrics that mattered.
The timeline. What happened when, including unexpected events.
The customer-visible effects. Tickets, NPS, error rates from the customer's perspective.

Walking into a PIR with this data lets the meeting focus on interpretation and action, not on assembling facts.

The Structure That Works

A 60-90 minute PIR with the following agenda:

1. Recap the goal (5 min)

The facilitator restates what the change was trying to achieve, in the original words. This is grounding; the team has been heads-down on execution and may have lost the original framing.

2. Walk the timeline (15 min)

What happened, in order. The deploy went out at T+0, the first issue appeared at T+47, we noticed at T+52, we rolled back at T+1:18.

The discipline: stick to what happened. Avoid jumping to "we should have" until later.

3. Assess outcomes (15 min)

Did the change achieve its goal? Match outcomes against the success criteria. Honest answers: yes, no, partially, too early to tell.

This is the section where teams most often fudge. The change shipped, customers didn't complain loudly, so we mark it a success. Resist the framing. If the original goal was "reduce p99 latency by 30%" and you reduced it by 8%, that's not a success — it's a partial outcome that warrants understanding.

4. What worked (10 min)

Specific things that contributed to the change going well. Tools, processes, decisions, people.

The point isn't celebration. It's identifying patterns to reproduce. "The pre-flight checklist caught two issues we'd have shipped without it" is reusable. "Everyone did a great job" is not.

5. What didn't (15 min)

Specific things that contributed to problems. Same discipline: name the thing, not the person.

If the team has a culture of blame, this section will produce platitudes. If they don't, it produces real signal — runbooks that were missing, dependencies that were undocumented, communication that didn't reach the right people.

6. Action items (15 min)

What we'll do differently. Each action item must have:

An owner. Specific person.
A description specific enough to verify. "Improve documentation" is not an action item. "Update the deployment runbook to include the database failover step we missed, by next Friday" is.
A due date.

If you can't write three good action items, the PIR didn't surface enough. If you write 25, the PIR is unfocused. Three to five is the sweet spot.

7. Capture and share (5 min)

The facilitator captures the output and circulates it to relevant teams. The PIR document is short — a page or two. Long documents don't get read.

What to Avoid

The blame search. Asking "whose fault was this" produces defensive answers and hides information. Ask what about the system or process allowed the problem to happen.

The vague action item. "Improve communication," "be more careful," "do better testing." Useless. Specific or skip it.

The action item without owner. "Someone should update the runbook." Nobody will.

The aspirational learning. "We learned that we should X" — was that actually a new insight, or just something everyone already knew but didn't do?

The post-success skip. Successful changes deserve PIRs too. Successful changes are where you learn what to reproduce. Skipping them means you only learn from failure.

The repeat finding. When the same finding appears in three consecutive PIRs, the issue is that you're not actually changing behavior. The PIR has become documentation, not improvement.

Who Should Be in the Room

A working PIR has:

The change owner or lead engineer
Key contributors — 2-3 people, not the full team
Affected operations — usually the on-call who handled any issues
A facilitator — ideally not the change owner. Different perspective.

Optional, based on the change:

A product owner if the change had product implications
A support representative if customer-facing
A neutral peer from another team, for fresh perspective

What doesn't help: full-team attendance. Engineers who weren't involved in the change get little from the meeting and clutter the conversation.

The Async Alternative

For lighter changes, a synchronous PIR is overkill. An async pattern:

The change owner writes a short PIR document (one page)
Specific named reviewers (2-3) comment within a week
Action items are extracted and tracked
The document goes in a searchable repository for future reference

This pattern scales better and captures most of the value with less time investment.

The PIR Repository

PIRs are most valuable in aggregate. A single PIR teaches one lesson; 50 PIRs teach patterns.

Keep PIRs searchable. Tag them by component, change type, outcome category. When planning a new change, search for similar prior changes and read their PIRs. The team's accumulated PIRs become an institutional memory that survives turnover.

A simple structure: PIRs in a wiki or repo, named by change and date, tagged with metadata. Periodically (quarterly?) review the corpus for patterns: which components show up repeatedly? Which findings recur? These are systemic issues worth addressing.

Tracking Action Items

The biggest PIR failure is action items that disappear after the meeting.

A working pattern:

Action items go into the team's regular work tracking system (Jira, etc.)
They're tagged as PIR-originated
They're reviewed at a regular cadence (monthly?)
Outstanding items are flagged

If action items aren't tracked, the PIR is theater. The work to track them is small; the benefit is real.

Distinguishing PIR from Post-Mortem

PIR is for any change worth reviewing. Post-mortem is specifically for incidents.

PIR asks: did this change work?
Post-mortem asks: why did this incident happen?

A change that caused an incident might warrant both — a post-mortem for the incident itself, a PIR for the change as a whole. The two have different framings and different participants.

In practice, many teams collapse the two when an incident is small. That's fine. Use whichever framing fits the conversation you need to have.

A Sample PIR Template

# PIR: [Change Name]
Date of change: [date]
Date of review: [date]
Facilitator: [name]
Attendees: [names]

## Goal
[What the change was trying to achieve, in original words]

## Success Criteria
[What we said would prove it worked]

## Outcomes
[Measured results against criteria]
Status: [Success / Partial / Failure / Too early]

## Timeline
[Key events with timestamps]

## What worked
- [Specific item with context]

## What didn't
- [Specific item with context]

## Action items
| Item | Owner | Due |
|------|-------|-----|
| [Specific action] | [Name] | [Date] |

Key Takeaway

A useful post-implementation review asks three questions: did the change achieve its purpose, what did we learn, what should we change. Time it after the immediate experience is fresh but before the effects are evaluable. Bring data into the meeting; use the time for interpretation, not assembly. Three to five specific action items with owners and due dates beats a long list of vague ones. Capture, share, and track. The point isn't the document; it's whether the team's behavior changes as a result. PIRs that produce no behavior change are theater.

ShiftQuality