Write a Bug Postmortem — Debugging Systematically, Part 9
Updated: Oct 4
Debugging Systematically · Part 9
The most expensive thing about an incident is having it twice. A post-mortem is how a team converts a bad day into a change that stops the same class of failure from recurring — but only if it hunts for the systemic cause instead of a person to blame. This walks through writing a blameless post-mortem: an honest timeline, the real root cause, and action items that actually get done.
The bug is fixed. Now: how do we prevent recurrence? A good postmortem turns an incident into learning.
Step 1: Blameless (5 min)
The first rule: blameless. Not "Pat made a mistake" but "the system allowed Pat's mistake to cause an outage."
Why:
Humans make mistakes; assume they will
Blame stops people from reporting issues
Systems are what we can improve
Bad: "Pat deployed without testing"
Good: "The deploy pipeline did not require a test stage before production"
Focus on the system; never the person.
Step 2: Timeline (10 min)
Time-stamped events:
14:32 UTC — Deploy v2.5.0 completed
14:33 UTC — Error rate spike began (5x baseline)
14:35 UTC — Oncall paged via Datadog alert
14:36 UTC — Oncall acknowledged
14:40 UTC — Identified failing endpoint
14:45 UTC — Initial mitigation: rolled back deploy
14:48 UTC — Error rate returned to baseline
15:15 UTC — Root cause identified
A linear log of what happened when.
Step 3: Impact (5 min)
What did users experience?
Duration of outage
Percentage of users affected
Revenue impact (if applicable)
Customer reports
SLO impact
Specific numbers, not "lots of users."
Impact: ~12% of POST /checkout requests failed
between 14:33 and 14:48 UTC (15 minutes)
Affected ~150 users; ~5 abandoned carts (~$300 revenue lost)
Step 4: Root Cause (10 min)
Not the surface cause; the underlying cause.
Surface: "the new code threw an exception." Deeper: "the new code assumed an env var was set; staging had it but production didn't." Even deeper: "we don't have a process for syncing env vars between staging and production."
Use "5 whys" — keep asking why until the system flaw is exposed.
Step 5: Contributing Factors (10 min)
Beyond root cause, what else helped?
"The deploy process doesn't require a test stage"
"Monitoring took 3 minutes to alert"
"Rollback was manual"
"No canary deploy"
Each is an opportunity to improve.
Step 6: What Went Well (5 min)
Don't only list problems:
"Alert fired within 3 minutes"
"Oncall responded within 1 minute"
"Rollback procedure worked on first try"
"Customer communication went out quickly"
Reinforces good practices. Builds team morale.
Step 7: Action Items (15 min)
Specific, owned, with deadlines:
| Action | Owner | Due | Status |
|--------|-------|-----|--------|
| Require integration test pass before deploy | Alice | 2027-02-15 | Open |
| Implement canary deploy | Bob | 2027-03-01 | Open |
| Auto-rollback on error spike | Carol | 2027-02-22 | Open |
| Document env var sync process | Dave | 2027-02-10 | Done |
Vague action items ("be more careful") never get done. Specific ones do.
Step 8: Distribute Widely (5 min)
A postmortem is useless if it's not read:
Email to the team
Post to engineering channel
Add to a searchable wiki
For major incidents: company-wide
Other teams learn. Patterns become visible (same root cause keeps appearing across teams).
Step 9: Follow Up (10 min)
Action items have deadlines. Track them:
Weekly review of open items
Specific assignees, not "the team"
Escalate if not progressing
Postmortems without follow-up = ritual. Same incident recurs.
Step 10: Template (10 min)
Standardize. A template helps:
# Postmortem: [Title]
**Date:** 2027-02-04
**Severity:** P1
**Author:** [Your name]
## Summary
[1-paragraph executive summary]
## Impact
- Duration:
- Users affected:
- Revenue impact:
## Timeline
[time-stamped events]
## Root Cause
[the underlying system flaw]
## Contributing Factors
- [each one]
## What Went Well
- [each one]
## Action Items
| Action | Owner | Due |
| ... |
Predictable structure. Easier to write; easier to read.
What You Just Did
A useful postmortem: blameless, with timeline, impact, root cause, contributing factors, what went well, specific action items, distribution, follow-up, template. Incidents become learning.
Common Failure Modes
Blame. Reporting drops; people hide bugs.
Vague action items. "Improve testing." → never done.
No timeline. Hard to learn or audit.
Postmortem skipped. Same incident in 3 months.
No follow-up. Action items languish.
Continue the Debugging Systematically path
Previous — Part 8: Debug Production Issues
For the reasoning behind this step-by-step guide, read Postmortem Template That Encourages Honesty.
Part of the Debugging Systematically learning path.
