Write a Bug Postmortem — Debugging Systematically, Part 9

Debugging Systematically · Part 9

The bug is fixed. Now: how do we prevent recurrence? A good postmortem turns an incident into learning.

Step 1: Blameless (5 min)

The first rule: blameless. Not "Pat made a mistake" but "the system allowed Pat's mistake to cause an outage."

Why:

• Humans make mistakes; assume they will

• Blame stops people from reporting issues

• Systems are what we can improve

Bad: "Pat deployed without testing"
Good: "The deploy pipeline did not require a test stage before production"

Focus on the system; never the person.

Step 2: Timeline (10 min)

Time-stamped events:

14:32 UTC — Deploy v2.5.0 completed
14:33 UTC — Error rate spike began (5x baseline)
14:35 UTC — Oncall paged via Datadog alert
14:36 UTC — Oncall acknowledged
14:40 UTC — Identified failing endpoint
14:45 UTC — Initial mitigation: rolled back deploy
14:48 UTC — Error rate returned to baseline
15:15 UTC — Root cause identified

A linear log of what happened when.

Step 3: Impact (5 min)

What did users experience?

• Duration of outage

• Percentage of users affected

• Revenue impact (if applicable)

• Customer reports

• SLO impact

Specific numbers, not "lots of users."

Impact: ~12% of POST /checkout requests failed
        between 14:33 and 14:48 UTC (15 minutes)
        Affected ~150 users; ~5 abandoned carts (~$300 revenue lost)

Step 4: Root Cause (10 min)

Not the surface cause; the underlying cause.

Surface: "the new code threw an exception." Deeper: "the new code assumed an env var was set; staging had it but production didn't." Even deeper: "we don't have a process for syncing env vars between staging and production."

Use "5 whys" — keep asking why until the system flaw is exposed.

Step 5: Contributing Factors (10 min)

Beyond root cause, what else helped?

• "The deploy process doesn't require a test stage"

• "Monitoring took 3 minutes to alert"

• "Rollback was manual"

• "No canary deploy"

Each is an opportunity to improve.

Step 6: What Went Well (5 min)

Don't only list problems:

• "Alert fired within 3 minutes"

• "Oncall responded within 1 minute"

• "Rollback procedure worked on first try"

• "Customer communication went out quickly"

Reinforces good practices. Builds team morale.

Step 7: Action Items (15 min)

Specific, owned, with deadlines:

| Action | Owner | Due | Status |
|--------|-------|-----|--------|
| Require integration test pass before deploy | Alice | 2027-02-15 | Open |
| Implement canary deploy | Bob | 2027-03-01 | Open |
| Auto-rollback on error spike | Carol | 2027-02-22 | Open |
| Document env var sync process | Dave | 2027-02-10 | Done |

Vague action items ("be more careful") never get done. Specific ones do.

Step 8: Distribute Widely (5 min)

A postmortem is useless if it's not read:

• Email to the team

• Post to engineering channel

• Add to a searchable wiki

• For major incidents: company-wide

Other teams learn. Patterns become visible (same root cause keeps appearing across teams).

Step 9: Follow Up (10 min)

Action items have deadlines. Track them:

• Weekly review of open items

• Specific assignees, not "the team"

• Escalate if not progressing

Postmortems without follow-up = ritual. Same incident recurs.

Step 10: Template (10 min)

Standardize. A template helps:

# Postmortem: [Title]
**Date:** 2027-02-04
**Severity:** P1
**Author:** [Your name]

## Summary
[1-paragraph executive summary]

## Impact
- Duration:
- Users affected:
- Revenue impact:

## Timeline
[time-stamped events]

## Root Cause
[the underlying system flaw]

## Contributing Factors
- [each one]

## What Went Well
- [each one]

## Action Items
| Action | Owner | Due |
| ... |

Predictable structure. Easier to write; easier to read.

What You Just Did

A useful postmortem: blameless, with timeline, impact, root cause, contributing factors, what went well, specific action items, distribution, follow-up, template. Incidents become learning.

Common Failure Modes

Blame. Reporting drops; people hide bugs.

Vague action items. "Improve testing." → never done.

No timeline. Hard to learn or audit.

Postmortem skipped. Same incident in 3 months.

No follow-up. Action items languish.

---

Continue the Debugging Systematically path

• Previous — Part 8: Debug Production Issues

• Next — Part 10: Build Debugging Tools Into Code

Part of the Debugging Systematically learning path.