From Blame to Learning: Incident Analysis That Drives Change
Updated: Oct 17
A postmortem can avoid naming anyone and still be built around one person's mistake. You can tell from the action items.
An engineer on the payments team changed one value in a configuration file: the payment gateway timeout, from 3000 to 30. The neighbouring settings in that file were in seconds; this one was in milliseconds. Every call to the gateway now timed out after 30 milliseconds, and for 45 minutes at evening peak almost no payment succeeded.
The postmortem was a model of blameless practice. Nobody was named, the facilitator opened by saying the meeting was about systems, not people, and the timeline was careful. It produced four action items:
Require a second reviewer for changes to payment configuration.
Add a comment in the file stating each setting's unit.
Remind engineers to test configuration changes in staging before production.
Shorten the payment-error alert window from 10 minutes to 2.
All four were done within a month. Three months later the shipping team changed a retry setting in their own configuration, it applied to every instance at once, and the label printer integration went down for 50 minutes. Its postmortem found the same shape of failure, and the payments team's fixes had changed nothing for shipping. (Composite example drawn from patterns across several organisations; the services and figures are illustrative.)
Nobody had been blamed. But three of the four fixes were aimed at the person who made the change: watch them more closely, tell them more, remind them harder. That is the failure this article is about, and it survives even in teams that run postmortems well.
Why blameless analysis still produces blame-shaped fixes
Google's SRE book sets the standard most teams aim for: a blameless postmortem "must focus on identifying the contributing causes of the incident without indicting any individual or team," and it assumes "everyone involved in an incident had good intentions and did the right thing with the information they had." The same chapter puts the reason in one line: "You can't 'fix' people, but you can fix systems and processes."
Teams adopt the language faster than the method. The tone of the meeting changes; the shape of the analysis doesn't. The analysis walks back from the outage until it reaches a human action that could have gone differently (the engineer typed 30), and stops there, because that action feels like an explanation. Everything after that point is about the action: how to stop someone making it again. A reviewer to catch it, a comment to warn against it, a reminder to test it. The person isn't blamed, but the fixes are aimed at a person all the same.
Patient-safety research named this pattern years ago and built a tool against it. The RCA² guidance, published by the US National Patient Safety Foundation, ranks corrective actions by how much they depend on people remembering to do the right thing. Its weaker actions, the ones that "require more reliance on humans to remember to perform the task correctly", are double checks, warnings, new procedures or policies, and training. Three of the payments team's four items sit exactly there: a double check, a warning, a policy reminder.
The test you can run: take the action items from your last five postmortems and mark each one person (it changes what someone must notice, remember or do) or system (it changes what the system allows, catches or limits). If most are person, your postmortems are blameless in tone only.
Ask what made it look right
The move that gets the analysis past "the engineer typed 30" is a different question. Not why did they do that? but what made that look like the right thing to do? Sidney Dekker calls the underlying idea the local rationality principle: people do what makes sense to them given their goals, what they knew and what they were paying attention to at the time. If an action looks foolish afterwards, the analysis hasn't yet found what made it sensible beforehand.
Asked of the payments change, the question opens up the system:
What did the engineer see? A file where every other timeout was a small number in seconds (connect_timeout: 5, read_timeout: 10). The gateway setting was the only one in milliseconds, and nothing in its name said so.
What did the tooling show? The pull request diff: one number changed, 3000 to 30. No range, no unit, no warning.
What did the engineer expect to happen? A 30-second timeout, safer than the previous behaviour under the gateway's recent slowness, which was the reason for the change.
What would have told them otherwise? Staging, if it had been realistic. Its gateway stub answered in under 5 milliseconds, so a 30-millisecond timeout passed every test there.
What did the reviewer see? The same one-line diff. A second reviewer would have seen it too.
Every answer is a property of the system: an inconsistent naming convention, a diff with no units or ranges, a staging environment that couldn't show slowness. None of them was visible in a timeline that recorded only what happened.
The test: for every human action in your incident timeline, write one sentence that starts "This looked right because…". If you can't finish the sentence, you haven't talked to the person, or you didn't ask the right question.
A timeline of what people knew, not just what happened
Most incident timelines have one column: events in order. Add two more, what the person could see at that moment and what the system knew, and the gaps between them become the findings.
Time | What happened | What the on-call engineer could see | What the system knew |
18:02 | Config change merged and applied to all payment instances at once | Change deployed successfully | Config changes skip the canary that code changes go through |
18:03 | Gateway calls start timing out | Nothing yet | Error rate on payment calls jumped from under 1% to over 90% |
18:13 | First alert fires | "Payment errors elevated" | The alert used a 10-minute window, so the first page came after 10 minutes |
18:13–18:31 | Investigation; the recent code deploy is rolled back with no effect | Code deploy log shows one deploy at 17:40 | Config changes didn't appear in the deploy log |
18:31 | Someone checks the config repository history | The 18:02 change | It had been there all along |
18:47 | Config reverted; payments recover | Recovery | Revert was manual, through the config repository |
Read across the rows and the incident stops being one mistake. It becomes a chain: a change the system let through without checking units or range, applied to the whole fleet at once, detected after ten minutes, and invisible to the people looking for the cause for another eighteen. Of the 45 minutes, the typo itself took a second. The rest belongs to the system.
The test: in your last timeline, find the longest gap between what the system knew and what the responders could see. That gap is usually worth more than the trigger.
Map the barriers, not a single root cause
Complex incidents rarely have one root cause, and looking for one sends the analysis back to the person. A more useful structure asks which barriers existed between the change and the customer, and which of them failed. Four kinds cover most software incidents.
Barrier | Question | In the payments incident |
Prevent | What should have stopped the bad change from reaching production? | No validation of units or range on config; staging couldn't show slowness |
Limit | What should have kept the damage small? | Config applied to every instance at once; no canary |
Detect | What should have noticed quickly? | 10-minute alert window |
Recover | What should have made fixing it fast? | Config changes missing from the deploy log; manual revert |
The original action items covered prevent (weakly, through review and reminders) and detect. Nothing touched limit or recover, which accounted for most of the 45 minutes. And because the shipping team's configuration went through the same pipeline, the same gaps were waiting for them.
The test: sort your last incident's contributing factors into the four barriers. A list with nothing under limit or recover is usually a sign that the analysis stopped at the trigger.
Grade every action by strength
This is the step most incident processes skip, and it does more to stop incidents coming back than anything else in the analysis. Before the postmortem closes, grade each action item on the RCA² hierarchy, translated to software.
Strength | What it changes | Software examples |
Stronger: needs no one to remember | The system makes the failure impossible, or simplifies or standardises the process | Schema validation with units and ranges that blocks the deploy; config changes forced through the same canary as code; removing a manual step entirely |
Intermediate: still relies on people, but makes errors visible or less likely | Software enhancements, checklists, redundancy, standardised communication | A tighter alert window; config changes listed in the deploy log with one-click revert; a staging stub that injects realistic latency |
Weaker: relies on people remembering | Double checks, warnings, new policies, training | A second reviewer; a comment stating the unit; a reminder to test in staging; a training session |
Graded, the payments team's original four items were three weaker actions and one intermediate. Rewritten against the barrier map:
Prevent (stronger): every configuration setting gets a schema with its unit in the name (gateway_timeout_ms) and an allowed range; a value outside it fails the pipeline. A 30-millisecond gateway timeout can't deploy.
Limit (stronger): configuration changes go through the same canary as code: 5% of instances first, with an automatic halt on error rate.
Detect (intermediate): the payment-error alert moves to a 2-minute window on error rate.
Recover (intermediate): configuration changes appear in the deploy log beside code deploys, with a one-click revert.
Two stronger and two intermediate actions, each tied to a failed barrier, and none of them depends on the next engineer being more careful. The schema and the canary live in the shared pipeline, so they protect every team that deploys configuration, shipping included. In the composite, the platform team built them after the shipping incident, and configuration incidents of that shape stopped.
Weaker actions still have a place. Training helps when the knowledge truly is missing, and a review costs little. The rule worth adopting is narrower: no significant incident closes with only weaker actions. At least one action per incident should be intermediate or stronger.
The test: grade the action items from your last three significant incidents. Count how many incidents closed with nothing stronger than a review, a warning, a policy or training. Each of those is likely to come back.
Follow-through, and the repeat that tells you it didn't work
Action items from postmortems are famously left undone. Three habits handle most of it: the items live in the team's normal backlog, not in the postmortem document; open items are reviewed monthly, and anything open for two months is either escalated or explicitly dropped; and each item has one named owner.
The more useful signal comes from across incidents, not within one. Tag each postmortem's contributing factors by category (configuration change, dependency timeout, capacity, data migration) and count them across quarters. A category that keeps appearing means the actions taken against it were scoped to the instance, the same pattern the post-implementation review recurrence audit catches in change reviews. In the composite, two configuration incidents in two teams within three months was the signal that the fix belonged in the shared pipeline, not in either team's process.
The test: list the contributing-factor categories from the last four quarters of incidents. Any category appearing three or more times is a system problem that individual postmortems have been treating as separate events.
What leadership does in the first hour
Blameless analysis depends less on the meeting than on what happens before it. The first question a senior leader asks after an incident tells everyone what the analysis is for. "Who made the change?" tells them it's a search for a person, and the people closest to the work will say as little as they can, which costs the analysis the very information the local-rationality questions need.
Questions that keep the analysis open: What do we know about the impact? What's needed to recover? What did the people involved see at the time? Then, after recovery: What would have stopped this, limited it, caught it sooner or fixed it faster?
Blameless doesn't mean nobody is ever accountable. The just culture idea from safety-critical industries draws the line: an honest error or a reasonable decision that turned out wrong is a system problem; knowingly reckless behaviour, such as deliberately bypassing a safety control everyone knows matters, is a conduct problem and is handled separately from the postmortem. Most incidents are firmly on the first side of that line. Treating them as if they were on the second is what makes people stop talking.
The test: count the near-misses reported in your organisation last quarter, the changes that almost caused an incident and were caught. A count of zero rarely means nothing went wrong. It usually means people don't think reporting is safe, or worth the trouble.
How deep to go
Not every incident needs a full facilitated analysis. Match the depth to the impact and to how new the failure is.
Depth | What it involves | Best suited for | Warning signs |
Lightweight review (30 min, team only) | Timeline, barrier map, graded actions | Minor incidents; familiar failure modes | Used for an incident that touched customers or money |
Full facilitated analysis (1–2 hours, neutral facilitator) | The above, plus interviews using the local-rationality questions | Significant customer impact; a failure nobody understood at first | Facilitator is the manager of the people involved |
Cross-team analysis | Several teams, focused on shared infrastructure and recurring categories | A contributing-factor category recurring across teams | Each team's postmortem closed separately with team-level fixes |
The postmortem template that encourages honesty gives you the document for any of the three. And many of the barrier gaps can be found before an incident at all: a pre-mortem on a risky change asks the barrier questions in advance.
What to do this week
Take your last significant postmortem and do two things. Grade each of its action items as stronger, intermediate or weaker. Then sort its contributing factors into prevent, limit, detect and recover. If every action is weaker, or a barrier has nothing under it, write one new action that would have changed what the system allowed rather than what a person had to remember, and get it into the backlog with an owner.
Final takeaway
A postmortem is blameless when its fixes stop depending on the next person being more careful, not just when it names nobody. Ask what made the mistake look right, map which barriers failed, and grade every action by how little it relies on memory. The incidents that stop coming back are the ones whose fixes changed the system.
This completes the Quality at Organizational Scale learning path. You've covered quality as organizational practice, engineering effectiveness programs, chaos engineering, and blameless incident analysis. The throughline: quality at scale is about organizational learning — building systems and cultures that get better from experience, including experience with failure.
Sources
Lunney, J. and Lueder, S., "Postmortem Culture: Learning from Failure", chapter 15 of Site Reliability Engineering: How Google Runs Production Systems (O'Reilly, 2016): blameless postmortems focus on contributing causes "without indicting any individual or team"; "You can't 'fix' people, but you can fix systems and processes." https://sre.google/sre-book/postmortem-culture/
National Patient Safety Foundation, RCA²: Improving Root Cause Analyses and Actions to Prevent Harm (2015; version 2, January 2016), Figure 3, Action Hierarchy: stronger actions (architectural changes, forcing functions, simplified and standardised processes), intermediate actions (redundancy, software enhancements, checklists, standardised communication) and weaker actions (double checks, warnings, new procedures or policies, training), based on the VA National Center for Patient Safety. https://www.ihi.org/library/tools/rca2-improving-root-cause-analyses-and-actions-prevent-harm
Dekker, S., The Field Guide to Understanding 'Human Error', third edition (CRC Press, 2014): the local rationality principle and the problem of hindsight in incident analysis.
Related reading
Keep learning. This article is part of the Advanced Quality Engineering path in the ShiftQuality Learning Center. Take quality from a team chore to an organizational property.
