Build an AI Agent, Stage 9: A Workflow Where Every Failure Stops or Hands Over
A step that can fail and let the run carry on is a check in name only. Give every step in the agent's workflow a failure policy that stops the run or hands it to a person, put verification before every action, and inject a fault at each step to prove it.
The example in this series is a composite scenario, built from patterns common to requirements intake in regulated organisations. It is not a single real organisation, and its figures are illustrative. The code is real: every output block below was produced by running it.
In the Requirements Agent's first weeks, the grounding check was a nuisance. It compared every quoted span in a draft with the requirement it came from, and it failed on things that didn't matter: a curly quote mark where the requirement had a straight one, a double space, a line break in the middle of a sentence. Analysts were waiting on drafts. So the team made two small changes. The check would run after the draft was saved, so it couldn't hold up the queue, and a failure would log a warning instead of stopping the run.
The false alarms stopped bothering anyone. By the fourth month the warning log held more than three thousand entries, and nobody read it. One of those warnings was real. A draft proposed that a broker's request duplicated a requirement and quoted a sentence the requirement no longer contained; it had been reworded in the latest version. The check caught it, logged it and let the draft through. The analyst, seeing a quoted span, confirmed the relation.
Nothing in the agent's code was broken. The model did what it was asked; the check did its job. What failed was the workflow around them: the order of the steps, and what each one was allowed to do when it failed.
This is Stage 9 of Build an AI Agent From Scratch. Stage 8 made retrieval able to say "none". The blueprint from Stage 3 already listed the workflow's parts and six hand-over cases. This stage wires them into a workflow and tests what happens when each part fails.
Why "log it and carry on" feels safe
Softening a failure always has a local reason. A check that raises false alarms slows people down. A search that times out once a day shouldn't take the whole queue with it. A warning seems to keep the information without the cost: anyone who wants to know can look.
Nobody looks. Google's SRE book makes the point about paging engineers, and it applies to any signal people are meant to act on: "Every page should be actionable", and too many that aren't make people "second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a 'real' page that's masked by the noise" (Ewaschuk, 2016). A warning in a log is weaker still than a page; at least a page interrupts someone.
The right fix for the false alarms was to fix the check: normalise quote marks, whitespace and line breaks before comparing, so a failure means a real mismatch. Softening its failure instead moved the cost from the analysts' queue to the people who would later act on a wrong draft.
The discovery move is a walk through the workflow with the people who own each failure. For every step, ask: if this step fails at three in the morning, who finds out, and what happens to the request? The team asked it about the grounding check and the honest answer was "nobody, and the draft is saved anyway". The same question exposed the search step, which, before Stage 5's fix, carried on with no results.
The test: list every place your agent's workflow catches an error. For each, write down who is told and what happens to the work in progress. Any answer of "it's logged" means the failure is invisible.
Code owns the sequence; the model owns judgement
Anthropic's guide to building agents draws a line that helps here. Workflows are "systems where LLMs and tools are orchestrated through predefined code paths"; agents are "systems where LLMs dynamically direct their own processes and tool usage" (Anthropic, 2024). Stage 2 found that one of the intake workflow's eleven steps needs an agent: finding related requirements, which by Stage 6 had become the agent's two skills, proposing relations and drafting questions. Everything around that one step, the order, the checks and the routing, can be code, and should be, because code can be tested step by step.
The reviewed workflow has eight steps, each of a kind the agentic workflows article describes:
Step | Kind | On failure |
intake | code | stop |
screen: regulatory signal, text addressed to the agent, more than one product line | decision | stop; routes to H1, H4 or H5 |
search | tool | retry once, then hand over (H3) |
read | tool | retry once, then hand over (H3) |
propose relations and questions | agent (the model) | stop |
verify: grounding check on every quoted span | verification | redo the proposal once with the check's message, then hand over (H6) |
route: conflict with a regulatory obligation | decision | stop; routes to H2 |
act: save the proposal and the question draft | action | stop, and remove anything saved in this run |
Three rules shape it. Every failure policy comes from a short list, retry once, redo, hand over, or stop, and "continue" and "warn" aren't on it. Every action comes after verification and routing, so nothing is saved that hasn't been checked. And a run that doesn't complete removes the drafts it saved, which is the rollback the blueprint's kill switch already describes, applied to a single run.
The test: draw your agent's workflow as a list of steps. Mark each one code, tool, model, verification, decision or action. If an action appears before the verification that is meant to check it, the check can only report damage, not prevent it.
One fault per step, before and after
The checker for this stage, workflow_check.py, does what the earlier checkers did: it reads a definition and checks it against the blueprint, then runs it. Every tool a step uses must be an allowed action in the blueprint; every action must come after a verify step; no failure policy may be "continue" or "warn"; every one of the blueprint's six hand-over cases must be reachable; and stopped runs must remove what they saved. Then it injects eleven faults, one per failure policy and per hand-over case, and checks where each run ends and what it leaves behind.
The step implementations are deterministic stand-ins: a search that times out on cue, a model step that produces a misquoted draft on cue. The workflow engine is real, and it runs the definition as written; there is no separate code path to drift from it.
The pilot workflow:
$ python workflow_check.py requirements-agent.blueprint.json v1
workflow v1: 6 steps (intake > search > read > propose > act > verify)
FAIL search on failure: continue
FAIL read on failure: continue
FAIL act runs before verify: a draft is saved before its quotes are checked
FAIL verify on failure: warn
FAIL hand-overs never reached: H1, H2, H3, H4, H5, H6 (of 6 in the blueprint)
FAIL compensation drafts saved before a stop are left behind
definition: FAIL (6 fail)
fault injection: 11 scenarios
PASS no fault completed, 2 saved
FAIL search times out once completed, 2 saved (a step failed and the run carried on)
FAIL search times out twice completed, 2 saved (expected handed_over:H3, 0 saved)
FAIL requirement unreadable twice completed, 2 saved (expected handed_over:H3, 0 saved)
FAIL one misquoted draft completed, 2 saved (a step failed and the run carried on)
FAIL two misquoted drafts completed, 2 saved (expected handed_over:H6, 0 saved)
FAIL regulatory conflict completed, 2 saved (expected handed_over:H2, 0 saved)
FAIL unflagged regulatory signal completed, 2 saved (expected handed_over:H1, 0 saved)
FAIL text addressed to agent completed, 2 saved (expected handed_over:H4, 0 saved)
FAIL spans two product lines completed, 2 saved (expected handed_over:H5, 0 saved)
FAIL saving a draft fails stopped, 1 saved (expected stopped, 0 saved)
verdict: FAIL (10 scenario(s) fail)
Ten of eleven runs end "completed" or leave a draft behind. The "one misquoted draft" line is the incident: the check failed, the run carried on, two drafts were saved. Notice that it has the same outcome and the same number of saved drafts as the no-fault run. Judged by outcome alone, it would pass; the checker fails it because a step failed and nothing recovered. The six hand-over cases that the blueprint wrote down in Stage 3 were never wired to anything.
The reviewed workflow:
$ python workflow_check.py requirements-agent.blueprint.json v2
workflow v2: 8 steps (intake > screen > search > read > propose > verify > route > act)
definition: PASS (0 fail)
fault injection: 11 scenarios
PASS no fault completed, 2 saved
PASS search times out once completed, 2 saved
PASS search times out twice handed_over:H3, 0 saved
PASS requirement unreadable twice handed_over:H3, 0 saved
PASS one misquoted draft completed, 2 saved
PASS two misquoted drafts handed_over:H6, 0 saved
PASS regulatory conflict handed_over:H2, 0 saved
PASS unflagged regulatory signal handed_over:H1, 0 saved
PASS text addressed to agent handed_over:H4, 0 saved
PASS spans two product lines handed_over:H5, 0 saved
PASS saving a draft fails stopped, 0 saved
verdict: PASS (0 scenario(s) fail)
Two faults recover: a single timeout is retried, and a single misquoted draft is redone with the check's message, which is what the check's message is for. Eight hand over to the person the blueprint named for that case. The last one, a failure while saving, stops the run and removes the proposal it had already saved, so an analyst never sees half a result.
The checker's own rules are tested by breaking the reviewed definition in one place: an action moved before verification, a failure policy softened to "continue", a hand-over route deleted. Further tests replay the incident on the pilot workflow and confirm that a stopped run leaves nothing behind. With seven new tests, the series suite has 65, and all pass.
The test: inject one fault into the step you trust most, and check where the run ends and what it left behind. If you can't inject a fault without editing the step's code, you can't test its failure policy.
Designing the workflow: a method
Mark the one place that needs judgement. Done when every other step is code, a tool call, a verification or a decision, and the model is called only where Stage 2 said judgement is needed.
Order the steps so every action follows its checks. Done when verification and routing come before any action they protect.
Give every step a failure policy from the short list. Retry once, redo, hand over to a named case, or stop. Done when no step can continue or only warn after a failure.
Wire every hand-over case. Done when each case in the blueprint is reachable from some step, and each hand-over carries the packet the blueprint specifies.
Inject one fault per policy and per case. Done when each scenario asserts the outcome and what is left saved, and runs in CI against the same definition the agent runs.
Tradeoffs and exceptions
Approach | Gains | Costs | Best suited for | Warning signs |
The model decides every next step | Flexible on open-ended tasks | Order and failure handling aren't testable step by step | Exploratory tasks with a person watching | Verification the model can choose to skip |
Fixed pipeline, soft failures | Simple; never blocks the queue | Failures pass silently | Nothing that saves or sends | A warning log nobody reads |
Workflow with hard failure policies | Every failure ends somewhere a person sees | More hand-overs at first; policies to maintain | Agents that save drafts others act on | Hand-over volume rising without a fix upstream |
Workflow plus a bounded agent step | Judgement where needed; control elsewhere | The agent step needs its own limits | Most production agents | An agent step that grows to absorb the workflow |
Hard failure policies cost something at first. When the team made the grounding check stop the run, hand-overs rose for two weeks, until the check's normalisation fixed the false alarms. That rise is information: each hand-over is a real mismatch or a real fault in the check, and either is worth knowing. If a hand-over case fires constantly, the fix is upstream, in the step that fails, not in the failure policy.
Anthropic's guide recommends "finding the simplest solution possible, and only increasing complexity when needed", and notes that agents commonly include stopping conditions "to maintain control" (Anthropic, 2024). A workflow with one agent step and hard failure policies is usually that simplest solution.
How Stage 9 goes wrong
Retry storms. A retry policy without a limit turns a slow dependency into a dead one. Retry once; then hand over. Draft actions that might be retried need the idempotency key from Stage 5.
Compensation that fails silently. Removing a run's drafts is itself a step that can fail. If it does, that is a hand-over too, with the IDs of what was left.
A workflow on paper and another in code. The definition says verify comes first; the code was written separately and doesn't. Run the agent from the definition, as the checker does, so there is only one.
Hand-over packets nobody can act on. A hand-over that says "error" sends the work to a person without the context to do it. Each case's packet is in the blueprint; test that it is filled in. Handle Agent Failures covers the mechanics of retries and fallbacks.
What to do next
Walk your agent's workflow this week with whoever owns each step, and ask what happens when it fails at three in the morning. Change any "continue" or "warn" to a hand-over or a stop, and write one injected-fault test for each. Stage 10 sets the human approval gates, each with one question, and tests them with planted bad drafts to prove the gate catches them.
Final takeaway
A check whose failure lets the run carry on is a check in name only. Let code own the order of steps, keep the model to the one step that needs judgement, put verification before every action, give every step a failure policy that retries once, redoes, hands over or stops, and inject a fault at each step to prove it. Then a failed check stops a bad draft instead of describing it in a log.
Sources
Anthropic, Building effective agents, 19 December 2024. Definitions of workflows and agents; "finding the simplest solution possible"; stopping conditions "to maintain control".
Rob Ewaschuk, Monitoring Distributed Systems, in Site Reliability Engineering, O'Reilly, 2016. "Every page should be actionable"; noise leads people to ignore "a 'real' page that's masked by the noise".
Series navigation. Previous: Stage 8: Retrieval That Can Say "None". Next: Stage 10: Human Approval Gates That Catch Planted Mistakes. The full stage list is on the Build an AI Agent From Scratch hub.
Frequently Asked Questions
What is the difference between an AI agent and an agentic workflow?
In a workflow, code decides the order of steps and what happens when one fails; the model is called for specific steps that need judgement. In an agent, the model decides which step comes next. Many production systems are mostly workflow with one or two agent steps inside.
How should an AI agent handle a failed step?
With a policy written into the workflow for that step: retry once, redo an earlier step with the failure message, hand the run to a named person, or stop. Carrying on after a failure, or logging a warning and carrying on, should not be allowed.
How do you test an AI agent's error handling?
Inject a failure at each step on purpose, one scenario per failure policy and per hand-over case, and assert that the run stops or hands over, and that nothing it saved before the failure is left behind.
Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.
