top of page

Incident Command Field Card

S. West
2 hours ago
5 min read

Four engineers joined the outage call, each with a different theory, and for forty minutes nobody was deciding anything.

Search had stopped returning results on a Sunday evening. One engineer scaled up the cluster, another reverted Friday's release, and a third restarted the cache, undoing the first fix. A director kept asking for an estimate and got a guess that proved wrong.

(Composite example, assembled from patterns common in teams that respond to outages without named roles. The figures are illustrative.)

None of those engineers lacked skill. What was missing was structure: who decides, who talks to the outside world, who works the problem. This card starts from that gap, giving each responder one role, a severity that sets the response, a timed checklist and fill-in updates. Print it and rehearse it.

Step 1: assign the roles

These roles come from the Incident Command System, which California's FIRESCOPE programme built after the 1970 wildfires and which FEMA's NIMS now uses. Google's SRE book and PagerDuty's incident response docs both adapt it for software.

Name each role when the incident opens. One person per role.

  • ☐ Incident Commander (IC): ______. Coordinates and decides; does not debug, write code or investigate. Holds the current picture, hands out tasks, breaks deadlocks and declares the incident resolved. Suits a senior engineer or manager who can stay out of the technical weeds.

  • ☐ Communications Lead: ______. Owns every outward update: status page, stakeholders, customers, executives. Keep this apart from the fixing.

  • ☐ Subject Matter Experts (SMEs): ______. Investigate and fix, reporting to the IC on the agreed channel.

  • ☐ Scribe (SEV-1 and SEV-2): ______. Keeps the timeline, so the IC doesn't have to.

Step 2: declare a severity that sets the response

Severity should tell everyone what happens next. When in doubt, pick the higher level; downgrading later costs little.

Level

Situation

Response it triggers

SEV-1 Critical

A core service is fully down for most or all customers; revenue is affected; data loss is possible or confirmed

Everyone joins, people get woken, executives are told at once

SEV-2 High

A core service is badly degraded, or a non-core service is fully down; workarounds fall short

On-call plus SMEs; stakeholders told within 30 minutes

SEV-3 Medium

Some customers or one feature partly degraded; a workaround exists

On-call in business hours; stakeholders told within 2 hours

SEV-4 Low

Small issue, little impact, no urgency

Normal ticket queue

The notification times are starting defaults. Set your own and write them down.

Declared severity: ☐ SEV-1 ☐ SEV-2 ☐ SEV-3 ☐ SEV-4

Step 3: work the first 15 minutes from the list (suggested timings)

  • ☐ 0 to 2 minutes, acknowledge and assess: what is broken, how many customers feel it, and is it spreading?

  • ☐ 2 to 5 minutes, declare and appoint: set the severity and name the IC. Unsure? Go higher.

  • ☐ 5 to 10 minutes, assemble: the IC pages the SMEs needed; the Communications Lead drafts the first notice.

  • ☐ 10 to 15 minutes, plan the investigation: the IC and SMEs list likely causes, split them into tracks and agree a reporting rhythm (every 10 to 15 minutes for SEV-1).

Step 4: send updates from templates

Stick to facts. Promise the next update time, not a fix time you can't back up.

Internal update

  • Incident: ______ Severity: ______

  • Status: ☐ investigating ☐ identified ☐ mitigating ☐ resolved

  • Impact: ______ Working theory: ______

  • Next actions: ______ Next update at: ______

Customer update

We're seeing a problem with [service]. Around [X]% of customers may notice [symptom]. Engineers are working on it, and our next update will be at [time].

Executive update

  • In one sentence: ______

  • Customers affected: ______

  • Business effect (revenue, reputation, contracts): ______

  • Status now: ______ Expected resolution (honest, or "still investigating"): ______

Step 5: escalate on severity and time, not on comfort

Example thresholds. Tune them to your service, but pick numbers before you need them.

  • ☐ SEV-1 still unmitigated after 30 minutes: bring in engineering leadership.

  • ☐ SEV-1 where data loss is confirmed: tell executives straight away.

  • ☐ Any incident running past 2 hours: brief leadership.

  • ☐ Any incident with regulatory consequences: tell legal.

Early escalation is the process working; thank people for it.

Step 6: close the loop

  • ☐ Run a blameless post-incident review within a few days of resolution (we aim for 48 hours, while memory is fresh). Cover the timeline, causes and contributing factors, what helped, what hindered, and actions.

  • ☐ Ask about the system ("what let this change through?"), not the person.

  • ☐ Give every action a named owner and a due date, and track it to closed. If few actions close, the reviews are producing lessons nobody applies.

Step 7: rehearse with game days

  • ☐ Break something realistic in a non-production environment, run the whole card end to end, then debrief.

  • ☐ Run game days at least quarterly (our default).

  • ☐ Rotate roles so every responder takes a turn as IC.

  • ☐ Include scenarios that need escalation and more than one team.

Culture check

  • ☐ People defer to the IC during an incident.

  • ☐ Reviews stay blameless in practice, not only on paper.

  • ☐ Incidents are treated as learning, not as verdicts on people.

  • ☐ Game-day time survives pressure to ship.

In our experience, the teams that handle incidents best aren't the ones with the best engineers. They're the ones with the best processes, practised, documented, and continuously improved.

How to use it

  • Who: the engineering lead who owns on-call, with two or three regular responders. Fill in the defaults once, then copy the card into your runbook.

  • Output: named roles, your own severity times, filled templates and written escalation thresholds.

  • Done when: a game day has run the card end to end, someone other than the usual lead has acted as IC, and every review action has an owner and a date.

Where this changes

A small team can merge roles: on a two-person rota the IC also handles communications, and SEV-3 and SEV-4 can become one level. Keep one rule: the person deciding is not also typing the fix. Regulated or large organisations usually add a deputy IC, separate internal and customer liaisons, and timelines kept as evidence.

The reasoning behind the card is in Incident Command for Software Teams. Hand the first responder a ready-made runbook, and practise the whole card with the Game Day Starter Kit.

Final takeaway

Incidents go badly when nobody is clearly in charge. Name the roles, agree your thresholds in advance, and rehearse the card until it feels routine.

Sources

bottom of page