Write Incident Reports — Technical Writing for Engineers, Part 7
Technical Writing for Engineers · Part 7
Eight months after a payments outage, a new engineer is asked to change the retry logic that caused it. They find the incident report. The summary says the outage was "caused by a misconfigured retry policy, resolved by rollback". The timeline lists times and actions with no links. The root cause section says the on-call engineer "failed to notice" an alert. The only action item still open reads "improve monitoring". The new engineer learns that something went wrong and that someone was blamed. They don't learn what the system did, why the people involved made the choices they made, or what they should be careful of now. The report recorded the incident. It didn't teach anything.
The facilitation, the meeting and the follow-up process matter, and other posts cover them: From Blame to Learning on running the analysis, and the Postmortem Template That Encourages Honesty for the structure. This tutorial is about the writing. Most reports are written the week of the incident, by people who were there, and read months later by people who weren't.
The working claim: an incident report is written for a reader who wasn't there, and every section should let that reader reconstruct what happened and why it made sense at the time.
Before you start
Gather the raw material before you write a sentence: chat logs, the paging history, dashboards captured at the time, deploy records and the incident channel export.
Note who will read the report: the team, other teams who depend on the system, leadership, and possibly customers. You'll write one internal report and adapt it.
Block time within a few days of the incident. Memory fades quickly, and the chat log won't hold what people were thinking.
Step 1: Write the summary for someone outside the team (15 min)
The summary is the only part most readers will read, so it has to stand alone. In four or five sentences, it should answer: what broke, who was affected and for how long, whether it's fixed, and what is changing as a result. Use the reader's terms, not the system's internal names. "Card payments failed for about one in eight customers for 40 minutes" means something to a director. "pay-svc retry storm against ledger-v2" means something only to the team.
Leave the explanation of causes for later sections. A summary that tries to explain the mechanism usually explains it badly and ends up naming a single cause, which is the trap Step 4 deals with.
Test you can run: give the summary alone to someone outside the team. Ask them four questions: what broke, who was affected, is it fixed, and what is changing? Anything they can't answer is missing from the summary.
Step 2: Make the timeline evidence, not memory (20 min)
A timeline written from memory compresses time and tidies decisions. Write it from the record instead. Each entry should have a time in UTC, what happened or what someone did, and a link to its source: the log line, the alert, the chat message, the deploy. Where the record doesn't show something, say so ("not recorded; recalled by the on-call engineer").
Include what people knew at each point, not just what they did. "14:41 — On-call sees elevated latency on the payments dashboard; error rate panel shows normal (the error panel excluded timeouts, see link)" tells the later reader why the obvious signal was missed. That detail is usually the most valuable sentence in the report.
Test you can run: go through your timeline and check that every entry links to a source. Mark every entry that says someone "realised", "noticed" or "decided" without a source, and either find the evidence or label it as recollection.
Step 3: Describe decisions from inside the moment (15 min)
Hindsight makes every wrong turn look obvious. Sidney Dekker calls the corrective the local rationality principle: people do what makes sense to them given their goals, what they are paying attention to and what they know at that moment. A report that ignores this ends up describing people as careless, which is rarely true and never useful, because the next person in the same position will make the same choice for the same reasons.
The language gives this away. Counterfactual phrases ("should have", "failed to", "could have prevented") describe what didn't happen, from a position nobody had at the time. Rewrite them as what the person saw and why their action made sense:
Instead of "the on-call engineer failed to notice the alert", write "the alert fired in a shared channel alongside 30 low-priority alerts that morning; it had no distinct severity, and the on-call engineer was working the latency page."
Instead of "the engineer should have checked the migration", write "the deploy checklist didn't include the migration step, and the pipeline didn't verify it."
The rewritten sentences point at things the organisation can change. The originals point at a person.
Test you can run: search your draft for "should have", "failed to", "neglected", "simply", "obviously" and "just". Rewrite each sentence to describe what the person knew and why the action made sense to them.
Step 4: List contributing factors, not a root cause (15 min)
Richard Cook's How Complex Systems Fail (1998) makes the point engineers keep relearning: catastrophe in a complex system needs several small failures to combine, and naming a single "root cause" is a social act, not a technical finding. John Allspaw's 2014 essay "The Infinite Hows" makes the related point about the Five Whys: asking "why?" in a chain tends to end at a person, while asking "how?" collects the many conditions that had to line up.
So write a section of contributing factors, each one a condition that made the incident possible or worse. A table works well:
Factor | How it contributed | Changed by |
Retry policy had no jitter or cap | Clients retried in sync and multiplied load | Action 1 |
Error dashboard excluded timeouts | The main signal looked normal for 11 minutes | Action 2 |
Alert shared a channel with low-priority noise | The page didn't stand out | Action 3 |
Rollback needed a manual approval at night | Mitigation took longer than the fix | Accepted for now; reviewed next quarter |
Test you can run: for each factor, ask whether the incident would have happened, or been as bad, without it. If the answer is "yes, exactly the same", it isn't a contributing factor. Then check that every remaining factor appears in the actions or is explicitly accepted.
Step 5: State impact in the reader's units, with the method (10 min)
Impact numbers get quoted, so they need to be right and their origin clear. Give duration, who was affected and how, in units the reader uses: customers, transactions, revenue at risk, error budget consumed. Next to each number, say how it was measured and how confident you are. "About 12% of checkout attempts failed between 14:33 and 15:13 UTC (from the payment gateway's error logs; excludes customers who abandoned before submitting)" is a number a later reader can trust and reuse.
Test you can run: check that every number in the impact section has a "measured from" note. Any number without one is either an estimate, which should say so, or a guess, which should be removed.
Step 6: Write actions that survive the quarter (15 min)
The report's actions are what the organisation will remember, so they need the same precision as the rest. Each action needs an owner (a named person, not a team), a date, the contributing factor it addresses, and how you'll know it worked. "Improve monitoring" fails every one of those tests. "Add timeouts to the payments error-rate panel and alert on them separately; owner: platform on-call lead; by 30 Nov; done when a synthetic timeout pages within two minutes" passes them all.
Google's Site Reliability Workbook (2018) notes that postmortems drive change only when they are written well, acted on and widely shared. The writing is the part you control on the day.
Test you can run: check every action against four questions: is there a named owner, a date, a linked factor, and a way to verify it? Rewrite any action that fails one.
Worked example: rewriting the payments report (composite scenario)
Context. A regional bank's payments team had written a one-page report for a 40-minute card outage. It named "misconfigured retry policy" as the root cause and said the on-call engineer "failed to notice" an alert. It had three actions, none with dates.
Decision. Before the review meeting, the incident lead rewrote the report using the steps above. The timeline was rebuilt from the channel export and paging history, with links. Counterfactual sentences were rewritten. The single root cause became four contributing factors.
Reasoning. The first version would have produced one action, "fix the retry policy", and one unfair conclusion about a person. The evidence showed the alert had fired in a noisy shared channel and the error dashboard had hidden timeouts. Neither was the on-call engineer's doing.
What happened. The review produced four actions instead of one, each tied to a factor, with owners and dates. The dashboard change was made in a week and caught a smaller timeout spike the following month. The on-call engineer, who had expected to be blamed, contributed the detail about the dashboard that became the most useful finding.
Lesson. The rewrite didn't change what had happened. It changed what the organisation learned from it.
Trade-offs to make deliberately
Choice | You gain | You risk |
Timeline built from the record | Accuracy, and what people knew when | More time to write the report |
Contributing factors instead of one cause | Actions that address the real conditions | A longer list leadership may find less tidy |
Removing counterfactual language | Candour in this and future incidents | Readers who want someone held accountable |
Separate internal and customer versions | Each audience gets the right depth | Two documents to keep consistent |
Common failure modes
A summary only insiders understand. Most readers stop there.
A timeline from memory. Time compresses and decisions look tidier than they were.
"Failed to" and "should have". They point at a person and hide the system.
A single root cause. One action, and the other conditions stay in place.
Actions with no owner or date. They are still open at the next incident.
Final takeaway
An incident report is written for someone who wasn't there and will read it months later. Give them a summary they can understand, a timeline built from evidence, decisions described as they looked at the time, the conditions that combined to cause the incident, impact numbers with their method, and actions specific enough to finish. Your next action: open your team's most recent incident report and run the search from Step 3. Count the counterfactual phrases, and rewrite the first three.
Sources
Cook, R. I. (1998). How Complex Systems Fail. Cognitive Technologies Laboratory, University of Chicago.
Allspaw, J. (2014). "The Infinite Hows (or, the Dangers of the Five Whys)." Kitchen Soap, 14 November 2014; republished on O'Reilly Radar.
Dekker, S. (2014). The Field Guide to Understanding 'Human Error', 3rd ed. Ashgate (now CRC Press).
Beyer, B., Murphy, N. R., Rensin, D. K., Kawahara, K. and Thorne, S. (eds.) (2018). The Site Reliability Workbook, Chapter 10: "Postmortem Culture: Learning from Failure." O'Reilly. sre.google/workbook/postmortem-culture.
Continue the Technical Writing for Engineers path
Previous — Part 6: Document an API
Part of the Technical Writing for Engineers learning path. Related: Incident Command for Software Teams.


