Build an AI Agent, Stage 14: Rehearse the Rollback Before the Canary Starts
An agent release is many versioned parts in several systems, and "redeploy the previous version" puts back only one of them. List every part the release changes and who can put it back, rehearse the rollback and the kill switch in the sandbox, judge the rehearsal by what the runs loaded, and start the canary only when it passes.
The example in this series is a composite scenario, built from patterns common to requirements intake in regulated organisations. It is not a single real organisation, and its figures are illustrative: the rehearsal records in this stage are composite, not measured. The code is real: every output block below was produced by running it.
Release 1.5 of the Requirements Agent was ready in the first week of October. Its question skill, version 2.4, had passed the Stage 11 suite and the Stage 12 promotion rule. The release plan was one page: a canary on the motor line for three days, then every line. Under "Rollback" it said: redeploy 1.4.
The BA practice lead, who can pull the agent's kill switch, had two reasons not to trust that line. The first was September: the rollback of question skill 2.3 had been deployed at 11:00 on a Monday and, as Stage 13 found from the runs, had not taken effect on one worker until it was restarted two days later. The second was older. Since Stage 3, the blueprint checker had printed the same warning on every run: kill switch never rehearsed; rehearse it before release.
So the team rehearsed in the sandbox before the canary: release 1.5 out, rolled back to 1.4, then the kill switch pulled. The rehearsal failed three ways. The new prompt never reverted. The new question skill stayed on one worker for 31 minutes. And after the kill switch, the cleanup left behind a hand-over ticket the agent had created.
This is Stage 14 of Build an AI Agent From Scratch. Stage 13 built the run record that says what each run actually loaded. This stage uses it to prove a rollback works before any real request depends on it.
Why "redeploy the previous version" isn't a rollback plan
"Redeploy 1.4" assumes a release is one thing. An agent release is several: the agent's code, its prompt, each skill, the model it calls, the gateway policy that decides what its tools may do, the judge that scores it, the trace spec that records it. They live in different places, which is the agent version of the point in What a Release Actually Involves, Beyond Clicking Deploy. The code is in an image the pipeline deploys. The prompt is in a registry with an "active" pointer. The skills are bundles that workers cache. The gateway policy belongs to the platform team.
Redeploying the previous image puts back exactly what is in the image. Everything else stays where the release left it. And the result is worse than an incomplete rollback, because it is a new combination: the old agent with the new prompt, a pairing that was never in any release and never passed any test. A rollback that produces an untested release has traded a known risk for an unknown one.
The deploy log can't show any of this. It records what was deployed and when. Only the runs know what they loaded, which is why the run record from Stage 13 carries every component's version on every run.
The test: for your agent's last release, list every part that changed and the system each one lives in. Then read your rollback procedure and mark the parts it actually puts back. Every unmarked part will still be running after your next rollback.
Every part the release changes, and who can put it back
The discovery move is a question asked before the rollout plan is written: what does this release change, and who can put each one back? Asked of the 1.5 release, it turned a one-line rollback into a manifest, requirements-agent.release.v2.json, that lists eight versioned components, the previous version of each, its owner and its way back.
Four of the eight changed in 1.5: the agent, the prompt, the question skill and the gateway policy. The first three have owners and ways back on the agent team. The fourth was the awkward one. Gateway policy gw-8 is the change that made the gateway record a refused call differently from a failed one, the fix Stage 13 needed for its security reader. Rolling it back to gw-7 would undo that fix. Nobody on the agent team could roll it back anyway: the platform team owns it.
Some changes shouldn't go back, and the manifest has a word for them: forward-only. A forward-only component is acceptable only if it is proven compatible with the release it would be rolled back to. Here that is checkable from data: gw-8 must grant every scope that agent 1.4's tools use, which the checker reads from the blueprint. It grants all six.
The test: for every component your release changes, write down who can put it back and how. A component with no answer is either forward-only, and then needs a compatibility check, or it is a rollback you can't perform.
The rehearsal, judged by the runs
The checker for this stage, release_check.py, reads the manifest, the blueprint and a rehearsal record. It checks that every changed component has a way back, then judges the rollback the way Stage 13 taught: from what each run loaded after the rollback started, against an objective in the manifest that every run is back on 1.4 within five minutes. Then it checks the kill switch.
The first rehearsal, on the release as first written:
$ python release_check.py v1
MANIFEST
release 1.5 changes 4 of 8 components (rollback target 1.4, released before: yes)
agent 1.4.0 -> 1.5.0 back by: pipeline redeploys the previous image (agent team)
prompt 4.1 -> 4.2 back by: pipeline redeploys the previous image (agent team)
question_skill 2.2 -> 2.4 back by: skill bundle redeploy (agent team)
gateway_policy gw-7 -> gw-8 FAIL: no way back named (owner: platform team)
ROLLBACK REHEARSAL (sandbox, 2026-10-05)
rollback 1.5 -> 1.4 started 14:00; 47 runs after it; objective: every run on 1.4 within 5 min
agent reverted within 1 min 00 s (stale on w-2) PASS
prompt still 4.2 on the last run at 14:46: never reverted FAIL
question_skill reverted within 31 min 00 s (stale on w-2) FAIL
gateway_policy still gw-8 on the last run at 14:46: never reverted FAIL
runs on a combination never released: 46/47
KILL SWITCH
flag off at 15:00: 6/6 new requests to the manual queue PASS
2 runs in flight finished and made 3 drafts; cleanup removed 2, left 1 (SD-303 hand_over) FAIL
cleanup covers 2/3 draft actions in the blueprint; missing: hand_over FAIL
verdict: canary may NOT start; fix the failures and rehearse again
Each failure has a mechanism. The prompt never reverted because the manifest's way back for it was wrong: redeploying the image doesn't move the prompt registry's "active" pointer, and agent 1.4 read whatever the pointer said. The question skill took 31 minutes on worker w-2 because workers cache skill bundles for 30 minutes. That is the mechanism behind September's stale worker; the restart then had hidden it, not fixed it. And because of both, 46 of the 47 runs after the rollback ran on a combination of versions that had never been released.
The kill switch half worked. New requests went to the analysts' manual queue, but two runs already in flight finished after the flag and made three drafts. The cleanup script deleted the two question drafts and missed the hand-over ticket. It had been written from the blueprint's rollback sentence, "delete open proposals and drafts", and nobody had thought of a hand-over ticket as a draft. The blueprint's own action list does: hand_over is one of its three draft-class actions. The checker now derives the cleanup's coverage from that list rather than from the sentence.
Three fixes followed. The rollback script moves the prompt registry's pointer. Each run checks its loaded manifest id against the active one when it starts, and reloads if they differ, so a cached bundle can't outlive a rollback. The gateway policy is declared forward-only, with its compatibility checked. The cleanup covers every draft-class action. Then they rehearsed again:
$ python release_check.py v2
MANIFEST
release 1.5 changes 4 of 8 components (rollback target 1.4, released before: yes)
agent 1.4.0 -> 1.5.0 back by: pipeline redeploys the previous image (agent team)
prompt 4.1 -> 4.2 back by: prompt registry: move the 'active' pointer back (included in the rollback script) (agent team)
question_skill 2.2 -> 2.4 back by: skill bundle redeploy; each run checks its loaded manifest id against the active one at start (agent team)
gateway_policy gw-7 -> gw-8 forward-only: gw-8 grants 6/6 scopes the previous agent uses: PASS
ROLLBACK REHEARSAL (sandbox, 2026-10-07)
rollback 1.5 -> 1.4 started 14:00; 47 runs after it; objective: every run on 1.4 within 5 min
agent reverted within 1 min 00 s (stale on w-2) PASS
prompt reverted within 1 min 00 s (stale on w-2) PASS
question_skill reverted within 1 min 00 s (stale on w-2) PASS
runs on a combination never released: 0/47
every run on 1.4 within 1 min 00 s: PASS
KILL SWITCH
flag off at 15:00: 6/6 new requests to the manual queue PASS
2 runs in flight finished and made 3 drafts; cleanup removed 3, left 0 PASS
cleanup covers 3/3 draft actions in the blueprint PASS
verdict: rollback rehearsed 2026-10-07; canary may start (motor line only, 3 working days)
record in the blueprint: kill_switch.last_tested = "2026-10-07, sandbox"
canary exit: Stage 11 suite says RELEASE; Stage 12 verdict PROMOTE; Stage 13 trace check PASS on the canary's runs; analyst edit rate no more than 3 points above the control; no must-never failure
(Both from stage14-output.txt. The rehearsal records are composite and illustrative; every time and count is computed by the script. Sandbox runs start once a minute, so times are to the minute.)
Every run that started from 14:01 on loaded release 1.4, with the forward-only gateway policy. No run in the rehearsal ran a combination nobody had released. And the date of the rehearsal goes into the blueprint's kill_switch.last_tested, which is the field the Stage 3 checker had been warning about; a test in the suite confirms that recording it clears the warning.
The test: rehearse your next rollback in the sandbox and answer one question from the run record, not the deploy log: what was the last run that loaded the version you rolled back from? If you can't answer it, you don't yet know whether your rollback works.
A canary that knows how to stop
Only now does the rollout start. The manifest describes it in three stages. Two days in shadow, where the agent runs on every line's requests and its drafts are recorded but not shown to analysts. Three working days of canary on the motor line, with the home and commercial lines as the control. Then every line.
The SRE Workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation", where "the part of the service that receives the change is 'the canary,' and the remainder of the service is 'the control'" (Warner and Davidovič, 2018). For an agent, the evaluation is everything the earlier stages built, and the prompt regression gate from Prompt Regression Testing in CI has already run before the shadow stage begins. The canary moves to every line only if the Stage 11 suite says RELEASE, the Stage 12 verdict is PROMOTE, the Stage 13 trace check passes on the canary's own runs, the analysts' edit rate on the motor line is no more than 3 points above the control, and no must-never rule has failed.
The same chapter asks the question this stage exists to answer: "If the new release is completely broken, how quickly can we detect and revert?" The canary's exit criteria are the detecting half. The rehearsal is the reverting half, and it has to be done before the canary, because a canary you can't stop is just a smaller full release.
The test: for each criterion that ends your canary, name the number that triggers a rollback and who decides. If any criterion is "we'll look at it", the canary has no stopping rule.
Shipping a release: a method
List every versioned part. Code, prompt, skills, model, gateway policy, judge, trace spec. Done when the manifest names each part's version, previous version, owner and way back.
Settle the forward-only parts. Done when every part that can't go back is proven compatible with the release it would be rolled back to.
Rehearse the rollback and the kill switch in the sandbox. Done when the run record shows every run on the previous release within the objective, no run on an unreleased combination, and nothing left after the cleanup.
Record the rehearsal. Done when the blueprint's kill_switch.last_tested carries the date, and the warning the blueprint checker has printed since Stage 3 is gone.
Roll out in stages. Done when shadow, canary and full each have a scope, a duration and exit criteria with numbers, and the canary starts only after the rehearsal passes.
Tradeoffs and exceptions
Approach | Gains | Costs | Best suited for | Warning signs |
Redeploy the previous image | Simple; one command | Puts back only what is in the image | Agents whose prompt and skills live in the image | A prompt registry or skill cache outside it |
Manifest with a way back per part | Every part has an owner and a path | Writing and keeping the manifest | Agents with several versioned parts | A part whose way back is "the pipeline" |
Forward-only with a compatibility check | Keeps fixes that shouldn't be undone | A check per forward-only part | Gateway and schema changes | Forward-only declared with no check |
Rehearsal judged by the runs | Proves the rollback before it is needed | An afternoon in the sandbox per release | Any agent whose output reaches people | Rehearsal judged by "the deploy succeeded" |
Not every release needs a full rehearsal. A release that changes only a prompt held in the image, with nothing cached and no forward-only parts, has a rollback the previous release already proved. Rehearse whenever the manifest's list of changed parts includes something the last rehearsal didn't cover. Rehearsals also expire: a new cache, a new registry or a new owner changes the way back, so the manifest's ways back are reviewed with every release.
How Stage 14 goes wrong
The rollback is one line. "Redeploy the previous version" restores the image and nothing else. Write the manifest, with a way back per part.
The rehearsal is judged by the deploy log. The deploy succeeded; the runs kept loading the old skill. Judge the rehearsal from the run record, against an objective in minutes.
A rollback creates a release nobody tested. Old code with a new prompt is a new combination. Count runs on unreleased combinations, and make the count zero.
The cleanup is written from a sentence. "Delete drafts" missed the hand-over ticket. Derive the cleanup from the blueprint's list of draft-class actions, so a new action can't be forgotten.
What to do next
Write down every versioned part of your agent's next release, with its owner and its way back. Then rehearse the rollback and the kill switch in the sandbox before the canary, and read the result from the run record. Stage 15 keeps the agent honest once it is live: drift signals, the triggers that demote it to a lower autonomy level, and a seeded drift that proves both work.
Final takeaway
An agent release is many parts in several systems, and a rollback is real only when every part goes back. List each part with its owner and way back, prove the forward-only ones compatible, and rehearse the rollback and the kill switch before the canary, judged by what the runs loaded. Then the canary has a stopping rule that has already worked once.
Sources
Alec Warner and Štěpán Davidovič, Canarying Releases, in The Site Reliability Workbook, O'Reilly, 2018. Canarying as "a partial and time-limited deployment of a change in a service and its evaluation"; canary and control; "how quickly can we detect and revert?"
Series navigation. Previous: Stage 13: Observability That Answers Four Readers From One Trace. Next: Stage 15: Monitoring That Demotes the Agent, Proven by Seeded Drift. The full stage list is on the Build an AI Agent From Scratch hub.
Frequently Asked Questions
How do you deploy an AI agent safely?
Version every part the release changes, not only the code: the prompt, the skills, the model, the tool permissions. Name who can put each part back, rehearse the rollback and the kill switch before any real traffic, and then roll out in stages, starting with a canary that is compared with the rest of the service.
Why rehearse a rollback before a canary?
A canary is only safe if you can stop it. Until a rollback has been rehearsed, you don't know whether every part of the release actually reverts, how long it takes, or whether the rollback itself produces a combination of versions nobody has tested.
What is a forward-only change in an AI agent release?
A part that should not be rolled back, such as a gateway policy that fixed how refusals are recorded. It is acceptable only if it is proven compatible with the previous release, for example by checking that it still grants every permission the previous agent uses.
Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.
