Rollback Readiness Card
The rollback ran as documented and the old version was back in four minutes. The incident lasted two more hours.
One Tuesday a regional healthcare provider's notification service released a changed message format. At 21:10 the error rate crossed the trigger and the on-call lead called the rollback. About forty thousand new-format messages were already queued, and the restored code could parse none of them, so consumers crashed, retried and requeued them. The fix took three days: tag each message with a format version and let consumers ignore unknown ones.
(Composite example, assembled from patterns common in teams whose rollback plans cover only the code. The provider is not a real organisation and the figures are illustrative.)
Many teams rehearse reverting the code and never check the other four things a rollback has to restore. This card starts from what the change leaves behind, then moves the hard decisions to design time, when they cost days rather than an incident. Use one card per change: at design review, in the change template and before release.
Step 1: scope it by what the change writes
A rollback has to cope with everything the new version did while it ran. Tick each row and record what is really there.
☐ | What has to come back | Prompt | Your answer |
☐ | Code | Can we redeploy the previous version? Everyone practises this half. | ______ |
☐ | Configuration | Does a setting live outside our deploy, such as a vendor toggle? Who reverts it? | ______ |
☐ | Data | Which rows get written, in what shape? Can the old code read them? | ______ |
☐ | In-flight work | What waits in queues, jobs or retries in the new format? | ______ |
☐ | Things that left the building | Which emails, webhooks or payments go out? How do we compensate? | ______ |
Test: list every row, message, file and outbound call your latest change produced. That list is your rollback scope.
Step 2: name the rung
"Which rung?" beats "is it risky?" because the rung can be checked. Stop at the first yes.
Sends, charges or hard-deletes something external? Rung 5.
Transforms existing values in place? Rung 4.
Writes anything in a shape the old code cannot read? Rung 3.
Adds schema the old code can ignore? Rung 2.
Stateless, no schema or data change? Rung 1.
None fit? Finish Step 1 and retry.
This decision path is a helper we derived from the ladder in Rolling Back a Change Safely. If the two ever disagree, the ladder table below wins.
Rung | Example | Rollback means | Required action | ☐ |
1. Trivially reversible | Stateless service | Redeploy the prior version | None | ☐ |
2. Reversible with care | New column the old code ignores | Revert code, keep schema | Stay additive | ☐ |
3. Reversible with data work | Column renamed or its type changed | Revert code, then undo schema | Split into two deploys | ☐ |
4. Hard to reverse | In-place value migration | Restore the migration snapshot | Take it and test the restore | ☐ |
5. One-way door | Message sent, card charged, hard delete | Compensation only | Decide before shipping | ☐ |
Rung: ______ Named independently by: ______ and ______
If the two names differ, stop and settle it: that disagreement is the review's most useful output. In our experience, changes that seem dangerous usually sit on rungs one or two.
Step 3: move it down one rung
Reversibility is a design choice. Tick each move that applies.
☐ | Move | What it buys | Cost |
☐ | Additive schema: add, deprecate, drop later (expand–contract, Parallel Change on martinfowler.com) | Rung 3 becomes rung 2 | One extra deploy |
☐ | Two-phase data: write both shapes, read the old, then switch the read | Phase-one rollback is a config change | Two deploys |
☐ | New behaviour behind a feature toggle (Hodgson), off by default | Anyone on call can flip it, with no deploy | In our view, the highest-leverage move |
☐ | Idempotent operations | Re-running during recovery is safe | ______ |
☐ | Versioned or drained queue | Queued work survives a revert | The composite's three-day fix |
Decision rule: if lowering the rung takes less than a day and anything shared is touched, spend the day.
Step 4: pre-decide the trigger
People deciding mid-incident least want a rollback to be needed, so decide now. All four parts are required.
☐ | Part | Fill in | Not acceptable |
☐ | Observable threshold | e.g. 5xx above 1% for three minutes: ______ | "If it looks bad" |
☐ | One named authority | ______ | A committee or "whoever's on" |
☐ | Fix-forward time-box | e.g. one attempt, ten minutes: ______ | Open-ended attempts |
☐ | Default | Roll back when in doubt | Debating it live |
Test: in your last incident, how long passed between rollback first being raised and it happening?
Step 5: check when rolling back is wrong
Tick any case that could apply and write the alternative.
☐ | Case | Why reverting hurts | Alternative |
☐ | Rollback is riskier than staying | A half-applied migration can revert into a state neither version understands | ______ |
☐ | A one-way door has closed | Reverting does not unsend anything | ______ |
☐ | Both systems live mid-migration | Users told they moved get moved back | ______ |
☐ | A dependency failed | Reverting adds variables and fixes nothing | ______ |
☐ | A truly understood one-line fix | Rare: many 3am one-liners turn out to be hypotheses | ______ |
☐ The release communication plan has a rollback message row: who is told, what, when and how. Many plans leave this row out.
Card summary
Field | Entry |
Change | ______ |
Rung | ______ |
Move applied | ______ |
Scope gaps found | ______ |
Threshold, authority, time-box | ______ |
Cases where rollback is wrong | ______ |
Rollback message owner | ______ |
This week: run a rollback
☐ Revert a recent rung two or three change in staging and time it.
☐ Note what only appears in practice: cached connections, queued work in the new shape, config your deploy never touches.
☐ Add one question to the change template: what rung is this on, and what single move would lower it?
How to use it
Who: the change author, a reviewer who knows the data and integrations, and the on-call owner for Step 4. In our experience a first card takes under an hour.
Output: a named rung, any move applied, a four-part trigger and the cases where rollback is wrong.
Done when: two people named the same rung, the trigger is complete, and the rollback has been run in staging.
Where this changes
A small team can use Steps 2 and 4 alone for rung one and two changes, and the full card when a change writes data or touches a queue. Regulated or larger organisations should attach the card to every change request, keep it as release evidence and link the trigger to their incident response runbook. Rehearse the trigger in a game day. The reasoning is in Rolling Back a Change Safely.
Final takeaway
A rollback restores more than code, and its reversibility is settled at design. Name the rung, lower it where a day allows, and write the trigger down before anyone is tired.
Sources
Danilo Sato, "Parallel Change" (martinfowler.com, 13 May 2014), on the expand–contract pattern. https://martinfowler.com/bliki/ParallelChange.html
Pete Hodgson, "Feature Toggles (aka Feature Flags)" (martinfowler.com, 2016, revised 9 October 2017). https://martinfowler.com/articles/feature-toggles.html
PagerDuty, Incident Response Documentation, on pre-decided response and incident roles. https://response.pagerduty.com/
