Rolling Back a Change Safely

Rollback is treated as a procedure you execute when things go wrong. It is better understood as a property you either built into the change or didn't, decided weeks before anyone needs it.

Aldermoor Health's notification service shipped a new message format on a Tuesday. By 21:10 the error rate was clearly wrong, the trigger fired, and the on-call lead called it.

The rollback worked exactly as documented. The previous version was serving traffic four minutes later.

The incident lasted another two hours. The queue held roughly forty thousand messages written in the new format, and the old version couldn't parse any of them. Every consumer that picked one up crashed and retried, which put the message back on the queue. The deploy had been reverted. The system had not.

The code is the easy half

"Roll back" is shorthand for restore a known-good state, and the code is only one of the things that has to come back:

• Code — reverted. This part is genuinely easy and it's the part everyone rehearses.

• Configuration — including config that took effect somewhere else. A feature toggled in a vendor dashboard doesn't revert when your deploy does.

• Data — rows written during the window, in a shape the old code may not understand.

• In-flight work — queues, scheduled jobs, retries, anything already accepted but not yet processed. Aldermoor's failure lived entirely here.

• Things that left the building — emails sent, webhooks delivered, payments taken. These don't come back at all.

That list is why rollback duration correlates so badly with deploy duration. The deploy touched one artifact; the rollback has to account for everything the artifact did while it was live.

The test you can run: for your last change, list what it wrote while it was running — rows, messages, files, outbound calls. That list, not the deploy, is the actual scope of your rollback.

The reversibility ladder decides everything else

Where a change sits on this ladder determines the plan, the review it needs, and how much caution the window deserves. It's a more useful question than "is this risky", because it has a checkable answer.

Rung	Example	Rollback is	What to do about it
Trivially reversible	Stateless service, no schema or data change	Redeploy the previous version. Minutes	Nothing. Ship it
Reversible with care	Additive schema change — new column, old code ignores it	Revert code, leave the schema	Keep it additive and this stays free
Reversible with data work	Renamed or re-typed column; old code can't read it	Revert code and undo the schema, in order	Split into two deploys instead
Hard to reverse	Migration that transformed values in place	Restore from a snapshot taken at the migration	Take the snapshot. Test the restore
One-way door	Email sent, payment taken, record hard-deleted	There is no rollback, only compensation	Decide before, because after is too late

Two things fall out of this. First, most changes that feel risky are on the top two rungs and need less ceremony than they get. Second, the ones that quietly sit on rung four are usually described in a change request as "low risk" — which is exactly the sentence that stops meaning anything.

The test: name the rung for your next change before you write anything else about it. If two people on the team name different rungs, that disagreement is the most valuable thing you'll learn this week.

You move down the ladder at design time, not at 3am

This is the part that actually matters, and it happens weeks before any incident. A change is not handed to you with a fixed reversibility — you choose it.

• Keep schema changes additive. Add a column, deprecate the old one, drop it a month later in its own change. Costs one extra deploy; moves you from rung three to rung two.

• Two-phase the data. Phase one writes both old and new and reads old. Phase two switches the read. Rolling back during phase one is a config flip, and the old data never stopped being maintained.

• Put new behaviour behind a flag, defaulted off. The single highest-leverage move available. Rollback becomes flipping a boolean, which is fast, needs no deploy, and can be done by someone who doesn't know the codebase.

• Make the operation idempotent so re-running it during recovery isn't a second failure.

• Drain or version the queue. Aldermoor's fix was a format version on every message and consumers that skip what they don't recognise. Three days of work that would have turned a two-hour incident into four minutes.

Every one of these costs something up front. That cost is the price of the rollback being real, and it's much lower than the cost of discovering the rollback isn't.

The test: for a change you're planning, ask what it would take to move it down one rung. If the answer is under a day, and the change touches anything shared, take the day.

The trigger is a decision made while calm

Rolling back under pressure is hard for reasons that have nothing to do with the technology. The team wants to fix forward. The author is invested. Time pressure makes thinking expensive, and the people deciding are the people who least want it to be true.

So the decision gets made in advance, by people who aren't under pressure, and written down:

• A specific, observable trigger. "5xx above 1% for three minutes." Not "if it looks bad" — that phrasing puts the whole decision on whoever is most tired.

• One named authority. Not a committee. Committees don't decide during incidents; they take turns describing the situation.

• A time-box on fixing forward. "One attempt, ten minutes, then we go back." This is the mechanism that converts an open-ended chase into a decision.

• A default. When in doubt, roll back. Fixing forward feels productive and frequently isn't, and every minute spent trying is a minute customers spend on the broken version.

The on-call engineer's job at 3am is to execute a decision that sober people already made, not to re-litigate it.

The test: read your last incident timeline and find the moment rollback was first mentioned versus the moment it happened. The gap between those two is what a pre-decided trigger removes.

When rolling back is the wrong move

The bias toward rolling back is a default, not a rule, and there are cases where it makes things worse:

• The rollback is riskier than the state you're in. Reverting a data migration that's half-applied can leave you somewhere neither version understands. A degraded system you can reason about beats an unknown one.

• The change is already a one-way door and the door is shut. Emails went out. Reverting the code doesn't unsend them, and it may make the follow-up harder.

• You're mid-migration with both systems live. Rolling back moves a population that's already been told it moved, which is a communication problem before it's a technical one.

• The failure isn't yours. A dependency is down; rolling back your change changes nothing except the number of variables in play.

• A fix forward is genuinely one line and genuinely understood. The honest version of this is rare — most "one line" fixes at 3am are hypotheses — but it does happen, and the time-box is what keeps it honest.

Whichever way it goes, the affected people need telling either way, and the rollback message is the row most communication plans don't have.

The test: for the change you're about to ship, ask what would make rolling back worse than staying. If you can't answer, you haven't thought about the state yet.

What to change this week

Don't write a rollback plan. Execute one.

Take a change you shipped recently on rung two or three, and roll it back in staging. Time it. You are looking for the three things that only appear when you actually do it: the cached connection that doesn't reset, the queued work in the new format, and the config that lives somewhere your deploy doesn't reach.

Then add one line to your change template: which rung is this, and what would move it down one?

Aldermoor's notification service is on rung one now. The message format carries a version, consumers skip what they don't recognise, and the last rollback took four minutes end to end — the same four minutes as before, except that this time the system came back with the code.