Build an AI Agent, Stage 10: Human Approval Gates That Catch Planted Mistakes

S. West

2 hours ago

10 min read

A human approval step proves nothing until you know what it catches. Give each gate one question and the evidence to answer it, make the reviewer answer that question rather than click "approve", and plant known-bad drafts among the real ones so the catch rate is measured, not assumed.

The example in this series is a composite scenario, built from patterns common to requirements intake in regulated organisations. It is not a single real organisation, and its figures are illustrative: the planted-draft records in this stage are composite, not measured. The code is real: every output block below was produced by running it.

After the misquoted draft in Stage 9, the workflow was fixed: the grounding check now stops a bad draft before it is saved. But the incident had a second part that the fix didn't touch. The draft had reached an analyst, and the analyst had confirmed it. The quoted sentence was right there on the screen. So the BA practice lead asked a question nobody had asked since the pilot began: if a bad draft gets past the checks, will the person at the gate catch it?

The review screen made the answer easy to guess. Every draft, whether it proposed a relation, drafted questions or handed the request over, appeared on one screen as a formatted note with two buttons, Approve and Reject. Analysts approved 97% of drafts. The median decision took eight seconds.

To find out, the team ran an audit. Over four weeks they planted forty drafts they knew were wrong among the real ones: quotes that didn't match the requirement, conflicts labelled as duplicates, questions that suggested scope the requester hadn't asked for. The analysts caught nine.

This is Stage 10 of Build an AI Agent From Scratch. The blueprint from Stage 3 named a human gate for each of the agent's draft actions, each with one question. The pilot had built one screen for all of them. This stage builds the gates the blueprint described and measures them.

Why "a human approves it" feels like enough

A human in the loop is the most reassuring line in an agent's design. It satisfies governance reviews, it appears in every vendor's architecture diagram, and it puts accountability with a named person. Once it exists, it is rarely tested, because testing it would mean asking whether colleagues are doing their jobs.

The research says to test it anyway. Automation bias, "the tendency to over-rely on automation", has been studied in many fields, and a systematic review in the medical informatics literature identified mitigators in both training and design, including "the position of advice on the screen" and "the provision of information versus recommendation" (Goddard, Roudsari and Wyatt, 2012). The EU AI Act now writes the concern into law for high-risk systems: people overseeing them must be enabled "to remain aware of the possible tendency of automatically relying or over-relying on the output" (EU AI Act, Article 14).

None of this means the analysts were careless. The screen asked them to approve a whole note, and an agent that is right most of the time teaches its reviewers that approving is right most of the time. Eight seconds is what a reasonable person spends on a decision they have learned is almost always yes.

The test: pull your agent's approval log for the last month. Work out the approval rate and the median time per decision. If approval is above 95% and decisions take seconds, you don't yet know whether the gate catches anything.

One gate, one question, the evidence to answer it

The blueprint already had the design. It listed a gate for each draft action with a single question, and the pilot had collapsed them into one. The reviewed gates follow the blueprint exactly:

Gate	Action	The one question	Evidence shown	Reviewer answers
relation	propose_relation	Does each quoted span appear in the request and the requirement, and does the relation follow from them?	The request span; the requirement span at the cited version; the requirement's current version and date; the relation proposed	Both spans match and the relation follows / a span doesn't match / the relation doesn't follow
questions	save_question_draft	Does any question suggest scope the requester did not ask for?	Each question beside the request span it clarifies	No question adds scope / a question adds scope (mark which)
hand-over	hand_over	Should the agent have stopped here, and does the packet say why?	The hand-over case and its packet	Right to stop, packet complete / shouldn't have stopped / packet missing something

Three design choices do the work. Each gate asks one question, so the reviewer knows what they are checking. Each shows the evidence for that question and nothing else: the relation gate puts the two quoted spans side by side with the requirement's version, instead of a formatted note that reads as finished. And each answer is an answer to the question, not a verdict on the draft, so a reviewer can't confirm a relation without saying that the spans match. That is the "information versus recommendation" distinction from the review above, applied to a screen.

The fourth gate in the blueprint, accepting a request, belongs to the product owner and isn't one of the agent's actions; it stays outside this stage.

The discovery move here was a short session with two analysts, watching them review ten real drafts on the old screen and asking, for each, what did you check before you approved? For most drafts the honest answer was "that it read sensibly". Nobody had compared a quoted span with the requirement, because the screen didn't show the requirement.

The test: for each approval screen your agent feeds, write down the one question the reviewer is answering. If you can't write it in one sentence, or the screen doesn't show the evidence to answer it, the reviewer is approving an impression.

Plant known-bad drafts, the way airports test screeners

Redesigning the gates is a hypothesis. Measuring them needs drafts whose right answer is known in advance, and the cleanest source is to make them.

Airport security has done this for decades. Threat image projection inserts "fictional threat images (FTIs) of actual pre-recorded threat items" into X-ray images of real bags, and the system records whether the screener spots them; it "increases attention and motivation of screeners and is often used to measure their detection performance" (Riz à Porta, Sterchi and Schwaninger, 2018). The screener can't tell a projected threat from a real one until they have made the call, and they are told afterwards.

The reviewed gates do the same. One draft in twenty at the relation and questions gates is a planted, known-bad draft, built from the blueprint's own failure modes: a misquoted span, a conflict proposed as a duplicate, a question that invents scope. Planted drafts are marked in the system, never leave the review screen, and the reviewer is told at once whether they caught it. The hand-over gate doesn't plant drafts in this version; its errors are rarer, and the blueprint's failure modes don't yet include a bad hand-over to build them from.

The checker for this stage, gate_check.py, checks the gate definitions against the blueprint and then judges each gate by its planted-draft records. The pilot's screen, with the audit's results:

$ python gate_check.py requirements-agent.blueprint.json v1
gates v1: review
  FAIL  review              one gate for 3 actions: propose_relation, save_question_draft, hand_over
  FAIL  review              answers are approve/reject: the reviewer never answers the question
  FAIL  review              shows the draft note only, not the evidence the question needs
  FAIL  review              plants no known-bad drafts in normal running, so nobody knows if it keeps working
  definition: FAIL (4 fail)
planted drafts (bar: caught >= 90%, 95% lower bound >= 0.75)
  FAIL  review     caught 9/40 (22%), lower bound 0.12, median 8 s per decision
         3/12  misquotes an existing requirement
         2/10  misses a conflict with an accepted requirement
         4/18  invents a requirement
  verdict: FAIL (1 gate(s) below the bar)

And the reviewed gates, after four weeks of planting:

$ python gate_check.py requirements-agent.blueprint.json v2
gates v2: relation, questions, hand-over
  definition: PASS (0 fail)
planted drafts (bar: caught >= 90%, 95% lower bound >= 0.75)
  PASS  relation   caught 29/30 (97%), lower bound 0.83, median 41 s per decision
        15/15  misquotes an existing requirement
        14/15  misses a conflict with an accepted requirement
  PASS  questions  caught 19/20 (95%), lower bound 0.76, median 35 s per decision
        19/20  invents a requirement
  verdict: PASS (0 gate(s) below the bar)

(Both from stage10-output.txt. The planted-draft records are composite and illustrative; the checks and statistics are computed by the script.)

The catch rate went from 22% to 95% and above, and decisions went from eight seconds to around forty. That slower time is the cost of the gate doing its job, and it is the number to defend when someone asks why review got slower.

Top: the pilot's single review screen against the reviewed gates. The pilot showed every draft as one formatted note with Approve and Reject buttons; analysts approved 97 percent in a median of eight seconds. The reviewed design has one gate per action, relation, questions and hand-over, each asking the blueprint's one question, showing only the evidence for it, such as both quoted spans side by side with the requirement's version, and requiring an answer to the question. Bottom: planted known-bad drafts, composite records. In a four-week audit of the old screen, analysts caught 9 of 40, 22 percent, with a 95 percent lower bound of 0.12. After the redesign, with one draft in twenty planted, the relation gate caught 29 of 30, lower bound 0.83, and the questions gate 19 of 20, lower bound 0.76, both above the bar of 90 percent with a lower bound of at least 0.75.

The test: plant five known-bad drafts at your agent's busiest gate next week, built from the failure modes you most fear. Count how many come back rejected, and for the right reason.

How many planted drafts prove a gate works

The bar has two parts for a reason. A gate that catches 5 of 5 planted drafts has a perfect catch rate and very little evidence: the 95% lower bound on its true catch rate, from the Wilson score interval the checker uses, is about 0.57. A test in the suite confirms that 5 of 5 fails the bar. Catching 19 of 20 gives a lower bound of about 0.76, which is why the questions gate passes only narrowly. To claim more, plant more.

This is the same arithmetic as sizing a test set for an evaluation, applied to people instead of a model; Evaluating AI Agents covers it. The practical rule for the Requirements Agent: at least 20 planted drafts per gate before anyone says the gate works, and a standing rate high enough to reach 20 again within a month, so a drop in attention shows up within weeks rather than after an incident.

Planting also needs rules of its own. Planted drafts must be impossible to send: they exist only on the review screen, and the action behind the gate refuses any draft carrying the planted mark. Reviewers are told about the programme, though not which drafts are planted, and get immediate feedback after each decision. And the results are about the gate, not the person; a gate below the bar is a design problem first.

With seven new tests, covering the definition rules, a missing gate, the five-of-five case and both periods' verdicts, the series suite has 72, and all pass.

The test: for each gate you plant at, compute the lower bound on its catch rate from the plants so far. If it is below the rate you would quote to an auditor, you need more plants before you quote it.

Building approval gates: a method

One gate per action, one question per gate. Take them from the blueprint. Done when no gate covers two actions and each question fits in one sentence.
Show the evidence the question needs, and only that. Done when a reviewer can answer the question from what is on the screen without opening another system.
Make the answer an answer. Done when every option answers the gate's question, and none is a bare approve or reject.
Plant known-bad drafts from the failure modes. Done when each gate on a draft action has a standing plant rate, plants can't leave the screen, and reviewers get immediate feedback.
Judge each gate against a bar with a lower bound. Done when each gate has at least 20 plants judged, and the catch rate and its lower bound are reported monthly beside the approval rate and decision time.

Tradeoffs and exceptions

Approach	Gains	Costs	Best suited for	Warning signs
One review screen, approve or reject	Fast; simple to build	Reviewers approve impressions	Low-stakes drafts nobody acts on	Approval near 100%, decisions in seconds
One gate per action, one question each	Reviewers check something specific	More screens; slower decisions	Drafts that others act on	Questions that grow into checklists
Gates plus planted drafts	A measured catch rate	Building plants; a feedback loop; reviewer time	Gates the agent's safety case relies on	Plants that look different from real drafts
Remove the gate, rely on automated checks	No reviewer cost	No judgement where judgement is needed	Actions with deterministic checks only	Removing a gate because it slows things down

Planted drafts have a realism problem, and the airport research found the same: projected images can look artificial. If planted drafts are easier to spot than real mistakes, the catch rate flatters the gate. Build plants from real past errors where you can, and rotate them.

Not every gate needs planting. The hand-over gate here doesn't plant yet, and a gate whose decision is fully checked by code doesn't need a person at all. Plant where the safety case depends on a person's judgement; Where Human Approval Belongs covers which actions need a person in the first place.

How Stage 10 goes wrong

The approval rate as the success metric. A rising approval rate looks like a better agent; it may be a more tired reviewer. Report the catch rate on planted drafts beside it, always.
Plants that teach the wrong lesson. If every planted draft is a misquote, reviewers learn to check quotes and nothing else. Draw plants from every failure mode in proportion to the harm each would do.
Feedback that feels like surveillance. Planting works when reviewers know it exists and see it as a check on the gate. Publish the gate's results; keep individual results with the reviewer.
The gate that quietly widens. A new draft type is added to an existing gate "because it's similar". It isn't; give it its own question, or it inherits a question that doesn't fit.

What to do next

Pull your approval log this week and compute the approval rate and median decision time for each gate. Then plant five known-bad drafts at the busiest one. Stage 11 assembles the full test suite, including boundary and adversarial layers, where the must-never cases have to pass on five of five runs.

Final takeaway

A human approval step is a control only if you know what it catches. Give each gate one question and the evidence to answer it, make the reviewer answer that question, and plant known-bad drafts so the catch rate is measured against a bar with a lower bound. Then "a person approves every draft" is a claim with numbers behind it.

Sources

Kate Goddard, Abdul Roudsari and Jeremy C. Wyatt, Automation bias: a systematic review of frequency, effect mediators, and mitigators, Journal of the American Medical Informatics Association, 2012. Automation bias as "the tendency to over-rely on automation"; mitigators including the position of advice on screen and information versus recommendation.
EU Artificial Intelligence Act, Article 14: Human oversight. Article 14(4)(b) on remaining aware of automation bias.
Robin Riz à Porta, Yanik Sterchi and Adrian Schwaninger, Examining Threat Image Projection Artifacts and Related Issues: A Rating Study, IEEE International Carnahan Conference on Security Technology, 2018. How threat image projection works and its use to measure screener detection performance.

Series navigation. Previous: Stage 9: A Workflow Where Every Failure Stops or Hands Over. Next: Stage 11: A Test Suite Where Must-Never Means Never. The full stage list is on the Build an AI Agent From Scratch hub.

Frequently Asked Questions

How do you make human review of AI agent output effective?

Give each gate one question, show only the evidence that question needs, and make the reviewer answer the question rather than click approve. Then plant known-bad drafts among the real ones and measure how many reviewers catch, so you know whether the gate works.

What is automation bias?

The tendency to over-rely on an automated system's output. In agent review it shows up as reviewers approving drafts quickly and almost always, including the ones that are wrong.

How many planted drafts do you need to test an approval gate?

More than a handful. Catching 5 of 5 planted drafts sounds perfect, but the 95% lower bound on that catch rate is only about 0.57. Catching 19 of 20 gives a lower bound of about 0.76, so plan for at least 20 per gate before claiming the gate works.

Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.