
Incident Response Runbook Template
Updated: Oct 18
Google's Site Reliability Engineering book makes two claims in its introduction that shape how a runbook should be written. First: "Humans add latency." Recording the best practices ahead of time in a playbook "produces roughly a 3x improvement in MTTR as compared to the strategy of 'winging it'" (MTTR being mean time to recovery). The hero who knows everything can get by; a practised on-call engineer with a playbook does much better. Second: "SRE has found that roughly 70% of outages are due to changes in a live system."
The first claim is why runbooks exist. The second is what most runbooks get wrong. They start with the symptom and go straight to fixes for the component that's complaining, when the most likely cause is something that changed in the last few hours, and the safest first fix is often to undo it.
This post gives a template built around both claims, the details that make a runbook usable at 3am by someone tired, and three tests that tell you whether yours actually work.
The template
# Runbook: [Specific incident type]
Last verified: YYYY-MM-DD (by: name, how: drill / real incident)
Owner: [team or role]
Severity: [when this is P0 / P1 / P2]
Linked from: [alert names that point here]
## Symptom
[What you see that means you're in the right runbook]
## First 5 minutes
1. What changed? Deploys, config, flags, infra, upstream status
2. [Specific check, with link or command]
3. [Specific check]
## Diagnosis
- If a change landed shortly before the symptom → Mitigation 0 (undo it)
- If [signal A] → Mitigation 1
- If [signal B] → Mitigation 2
- If nothing matches → Escalate
## Mitigation 0: Undo the recent change
1. [Exact rollback / flag-off / revert command]
2. [Verification]
Risk: [what rolling back can break] Stop if: [condition]
## Mitigation 1: [Cause]
1. [Exact command]
2. [Verification after this step]
Risk: [what this can make worse] Stop if: [condition] → Escalate
## Escalate
Page: [role in your paging tool] Channel: [channel]
Say: symptom, start time, what you tried, current state
## Verify
[Signals that show it's resolved, with thresholds]
## After
[Timeline, ticket for the real fix, update this runbook]
Two lines in this template are missing from most: the "What changed?" check at the top of the first five minutes, and the Risk / Stop if line under every mitigation. The rest of this post explains why each part is there.
Symptom: confirm you're in the right runbook
The first thing the on-call engineer reads should let them confirm they've opened the right document.
Vague: "Database issues."
Specific: "Checkout 5xx error rate above 1% for 5+ minutes; connection pool exhausted in checkout logs. Alert: CHECKOUT-5XX."
Name the alert, the metric and threshold, and the log line. If the on-call can't match what they see to the symptom in a few seconds, they're in the wrong runbook, or the runbook doesn't exist yet.
First five minutes: ask what changed before anything else
If roughly 70% of outages follow a change, then "what changed?" is the highest-value question in the first five minutes. Make it the first step, and make it fast: a link to the deploy history for the affected services, the feature-flag audit log, recent config changes, and the status pages of upstream providers. Note the time each change landed against the time the symptom started.
Then the orientation checks: the service dashboard, the error rate and volume, any active incident already declared. These aren't fixes. They narrow down which branch of the diagnosis to take.
The test: for each of your top five runbooks, does the first step tell the on-call where to see recent changes, in one click or one command? If they'd have to ask in a channel, add the link.
Diagnosis: branch on what you see
After triage, the runbook branches. Put "a change landed shortly before the symptom" first, because it's the most common case and has the most general fix.
- Deploy, config or flag change within ~2h before the symptom → Mitigation 0
- "connection pool exhausted" in logs → Mitigation 1
- 502/503 from the load balancer, pods restarting → Mitigation 2
- 504s, downstream latency high → Mitigation 3
- Upstream status page shows an incident → follow the provider runbook
- Nothing matches → Escalate
Each branch names an observable signal, not a theory. "Database is overloaded" is a conclusion. "connection pool exhausted in logs" is something a tired person can check.
Mitigations: exact steps, and what each one risks
Every mitigation gets numbered steps, exact commands that can be copied, and a check after each step. The original version of this template stopped there. Add two more things to each one.
Risk: what this action can make worse. Restarting every pod at once while the database is struggling can cause a burst of reconnections that makes it worse. Rolling back code can fail if a schema migration has already run. Failing over can lose writes still in flight.
Stop if: the condition under which the on-call should stop following the runbook and escalate. A runbook that only says what to do will be followed past the point where it's wrong.
## Mitigation 1: Connection pool exhausted
1. Confirm: curl -s checkout.internal/admin/db-stats | grep pool
2. Check database health on the DB dashboard (CPU, active connections)
3. If the DB is healthy: restart pods in batches, not all at once:
kubectl rollout restart deployment/checkout -n prod
(rolling restart: by default at most 25% of pods are unavailable at once)
4. Watch the error rate for 3 minutes
Risk: if the DB itself is saturated, restarts add reconnect load.
Stop if: DB CPU > 90% or errors rise after the first batch → Escalate
to db-oncall; do not keep restarting.
Rollback deserves its own care, because it's the most common mitigation and not always as safe as it looks. A rollback that takes four minutes can still leave an incident running for hours if newer data or queued messages can't be read by the old code. The Rollback Readiness Card is a one-page check for how reversible a change is before it ships, which is exactly the information Mitigation 0 depends on.
The test: read each mitigation in your runbooks and ask, "what could this step make worse?" Any mitigation without a written risk and a stop condition is relying on the on-call to work that out at 3am.
Escalate: name the person, the place and the message
"Escalate to the appropriate team" fails exactly when it's needed. Say who: a role in your paging tool ("page db-oncall"). Say where: the incident channel. And say what to tell them: the symptom, when it started, what's been tried and the current state, so the next person doesn't repeat the first fifteen minutes.
For incidents big enough to need more than one responder, roles matter as much as steps. When several people work on the same outage with nobody coordinating, they can undo each other's fixes. The Incident Command Field Card gives every responder a role, a channel and one person making the call.
Verify and after: evidence, then learning
Don't declare an incident over on instinct. Name the signals: error rate back under its normal level on the dashboard, the alert cleared, a test transaction passing, no new reports in the support channel.
Afterwards, capture the timeline while it's fresh, file a ticket for the underlying fix, and update the runbook with whatever was missing or wrong. Every real incident is the best test the runbook will get. From blame to learning covers how to turn the incident into fixes that change the system, and the postmortem template gives the document.

Three tests that tell you whether your runbooks work
A runbook that's never been run by anyone but its author is a hypothesis. These three tests turn it into evidence.
1. Coverage: does every paging alert lead to a runbook? List every alert that can wake someone up, and check whether it links directly to a runbook. An alert with no runbook is an incident that will be handled by improvising, which is the case the SRE book's threefold difference is about.
2. The cold drill: can someone who didn't write it follow it? Have an engineer who didn't write the runbook execute it against staging, or a simulated failure, while you watch without helping. Time it, and write down every point where they stop or ask a question. Each stop is a missing step, an assumed piece of knowledge or a broken command. The Game Day Starter Kit covers running these drills as a team exercise.
3. Freshness: when was it last verified, and how? The "Last verified" line should record a date, a name and how (a drill or a real incident). Treat anything not verified in the last six months as suspect: systems, commands and people change faster than runbooks.
(Composite.) A payments team runs all three tests. It has 31 paging alerts, and 12 link to a runbook. Of the 9 runbooks it has, 4 haven't been verified in over a year. The first cold drill, on the checkout error runbook, takes 34 minutes against an expected 10: the new engineer gets stuck at a dashboard link that no longer exists, a command that needs a permission on-call engineers don't have, and a restart step with no warning about database load. Over the next quarter the team writes runbooks for the 8 alerts that fired most often, adds "What changed?" and Risk / Stop if lines to all of them, and reruns the drill. It takes 11 minutes.
The test: run the coverage count this week; it takes an hour. Then schedule one cold drill on the runbook for your noisiest alert.
A worked example
# Runbook: Checkout 5xx error spike
Last verified: 2026-04-12 (by: payments on-call, how: drill in staging)
Owner: payments-oncall
Severity: P0 above 5% errors; P1 at 1-5%
Linked from: CHECKOUT-5XX
## Symptom
Checkout 5xx rate > 1% for 5+ minutes.
## First 5 minutes
1. What changed? Deploys: https://deploys.internal/checkout
Flags: https://flags.internal/audit?service=checkout
Upstream: payment provider and cloud status pages
2. Checkout dashboard: https://grafana.internal/d/checkout
(error rate, volume, latency by endpoint)
3. Check #incidents for an incident already declared
## Diagnosis
- Change within ~2h before the spike → Mitigation 0
- "connection pool exhausted" in logs → Mitigation 1
- 502/503, pods restarting → Mitigation 2
- 504s, payment-service latency high → Mitigation 3
- Nothing matches → Escalate
## Mitigation 0: Undo the recent change
1. Flag change: turn the flag off in the flag console
2. Deploy: ./scripts/rollback.sh checkout <previous-version>
3. Watch error rate for 5 minutes
Risk: rollback fails if a schema migration ran with the deploy.
Stop if: the deploy included a migration → Escalate before rolling back.
## Mitigation 1: Connection pool exhausted
(as in the section above)
## Mitigation 3: Downstream timeouts
1. Check payment-service and inventory-service latency panels
2. Page the slow service's on-call
3. Enable the circuit breaker: ./scripts/checkout-cb.sh on
Risk: with the breaker on, affected payments fail fast instead of
waiting; customers see an error sooner.
Stop if: error rate rises with the breaker on → turn it off, Escalate.
## Escalate
Page: checkout-senior-oncall Channel: #incident-active
Say: symptom, start time, changes found, steps tried, current state
## Verify
- 5xx rate < 0.1% for 10 minutes
- ./scripts/test-checkout.sh passes
- No new reports in #support-internal
## After
- Summary in #incidents; POSTMORTEM ticket if customer impact > 5 min
- Update this runbook: anything missing, wrong or slow
Per-incident and general runbooks
Most of your runbooks should be per-incident: one symptom, specific actions. They're the ones used most. A short general runbook sits above them: how to declare an incident, who takes command, where to communicate, and when to escalate. Incident command for software teams covers that layer.
What to do this week
List your paging alerts and mark which ones link to a runbook. For the alert that fires most often, check its runbook for three things: a "What changed?" step first, a Risk and Stop if line under every mitigation, and a "Last verified" line that says who, when and how. Fix what's missing, then book a 30-minute cold drill with someone who didn't write it.
Final takeaway
A practised responder with a playbook recovers much faster than one improvising, and most outages follow a change. So a runbook should ask what changed before anything else, give exact steps with a check after each, and say for every fix what it risks and when to stop. Then prove it works: every paging alert links to one, someone other than the author has run it, and the last verification is recent.
Sources
Beyer, B., Jones, C., Petoff, J. and Murphy, N. R. (eds.), Site Reliability Engineering: How Google Runs Production Systems (O'Reilly, 2016), Introduction: "Humans add latency"; a playbook "produces roughly a 3x improvement in MTTR as compared to the strategy of 'winging it'"; "roughly 70% of outages are due to changes in a live system." https://sre.google/sre-book/introduction/
Kubernetes documentation, "Deployments": rolling updates, where maxUnavailable and maxSurge each default to 25%. https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
Related reading
Keep learning. This article is part of the Advanced Quality Engineering path in the ShiftQuality Learning Center. Take quality from a team chore to an organizational property.
Part of the Engineering Templates & Playbooks guide — ShiftQuality's collection of ready-to-adapt templates.


