Agent Verification Checklist
The onboarding agent closed the ticket with "accounts created, access granted". On Monday morning, none of the three new starters could log in.
The directory service had answered {"status":"accepted"}, then rejected the request a minute later for a missing cost centre. The agent read "accepted" as finished. Every dashboard stayed green, and the gap surfaced only when a manager rang the service desk.
(Composite example, assembled from patterns common in teams running tool-calling agents in production. The figures are illustrative.)
The agent did not invent a fact; it invented a status. This worksheet gives your team the most reliable defence: external verification, backed by loud tools, honest prompts and a discrepancy rate. Prompting helps (Zhu et al. (2026) saw false-success reports fall from 22.8% at baseline to 0.8% with a structured evidence contract), but it does not replace checking the record. Start from the action end: list what the agent claims to change, then decide how to prove each claim.
Step 1: map your side-effecting actions
List every action the agent reports that changes state. For each, ask: if this failed unnoticed, would the damage justify the cost of a check? For actions that reach customers, money or records, it usually does.
Action the agent reports | Action type | System of record | Verify? |
"Created the user account" (composite) | Resource creation | Directory service | Y |
______ | ______ | ______ | ☐ Y ☐ N |
______ | ______ | ______ | ☐ Y ☐ N |
Step 2: pick a verification pattern for each action
Prove each claim from the system of record, never the agent's report. Tick one pattern per Step 1 row.
☐ Resource creation: look the resource up and compare its attributes with the intent (for "ticket raised in Jira", search Jira for the expected title and reporter).
☐ Communication: query the sending service's audit log to confirm the provider accepted the message (and its delivery status, where exposed).
☐ State changes: read the resource back and compare it with the claimed state.
☐ External calls with side effects: ask the receiving service whether it registered the call.
☐ Data writes: read the row or document back and compare stored values.
Rule of thumb: a match logs a verified completion; a mismatch raises an alert. If the system of record is eventually consistent, re-query after a short delay before alerting.
Step 3: audit each tool for loud failure
Structured results let an agent fail loudly; free text lets it fail quietly. Score every tool.
Property | What good looks like | Tool: ______ | Tool: ______ |
Explicit status | A top-level boolean or enum | ☐ | ☐ |
Structured errors | An error type and message, as data | ☐ | ☐ |
Queued is not done | Separate accepted, queued and done values | ☐ | ☐ |
Timeout handling | A typed timeout, never an empty reply | ☐ | ☐ |
Retry safety | Idempotency keys or conditional writes | ☐ | ☐ |
The MCP tools specification supports this: errors carry isError: true, results can follow an outputSchema, and clients should time out tool calls.
Then tick the silent-failure triggers each tool can produce:
☐ Timeouts, with silence taken as success
☐ Partial replies the agent completes itself
☐ Unfamiliar error formats, read as success
☐ Ambiguous statuses, such as {"status":"queued"} read as done
☐ Soft failures: a warning inside a normal reply
(ShiftQuality's working list, not an external taxonomy.)
Step 4: prompt for honest failure
Prompting lowers the rate but never removes it, so Step 2 still applies. Confirm the system prompt has each clause, in your own words:
☐ Uncertainty: "Treat an action as done only when a tool result proves it. Otherwise call it unconfirmed, then check or escalate."
☐ Observation: "After each tool call, note what the result said and what you concluded."
☐ Two-part report: "List the actions you attempted, then mark each confirmed or unconfirmed."
☐ Permission to stop: "If a call errors or its result is unclear, stop and report it."
Step 5: measure the discrepancy rate
Standard monitoring is blind to this: the agent reports completion, the reply is fast and no error fires. Track one more number:
Discrepancy rate = (tasks where agent says "completed" but verification says "failed") ÷ tasks verified
☐ Steps 1 and 2 are live; without them there is nothing to measure.
☐ The rate is plotted over time.
☐ A rising rate opens a review: it signals model drift, prompt drift, tool changes, or a verifier that has itself broken.
Week | Tasks verified | Said "completed", verification "failed" | Rate |
______ | ______ | ______ | ______ |
______ | ______ | ______ | ______ |
Step 6: score your coverage
☐ Every "Y" action in Step 1 has a Step 2 pattern. You can see the failure.
☐ Tools pass Step 3 and the prompt passes Step 4. You make it rarer.
☐ The Step 5 rate has four weeks of history. You catch drift early. (Four weeks is a working default, not a benchmark.)
Any "Y" action without a pattern is your first gap. A silent failure caught the same day is a manual fix; caught weeks later, it is that fix plus every decision built on the false report.
How to use it
Who: the engineer who owns the agent, with each system-of-record owner. Start with Step 1.
Output: an action map, a pattern per action, a tool audit, a prompt checklist and a weekly discrepancy rate.
Done when: every action worth verifying is checked against its record, and the rate is reviewed on a schedule.
Where this changes
A small team with one low-stakes agent can verify only the "Y" rows and review the rate monthly. Regulated teams, or agents that move money or personal data, should verify every side-effecting action, keep the log as audit evidence and alert on any mismatch.
Read the reasoning in the essay on agents that report success they never achieved. This checklist goes deep on modes 04 and 14 of the Agent Failure Mode Catalogue. For the code side, see the tutorial on handling agent failures and why per-step reliability compounds.
Final takeaway
An agent's report is a claim, not evidence. Check each claim against the system that holds the truth, make tools fail clearly, and watch the discrepancy rate so a quiet failure becomes a loud one.
Sources
Zhu, J., Xie, S., Chen, A. L. F., Ding, X. et al., Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models, arXiv preprint (September 2026): false-success rates of 22.8% at baseline and 0.8% with a structured evidence contract. https://arxiv.org/abs/2609.35732
Soni, H., ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents, arXiv preprint (6 July 2026), ICML 2026 AIWILD/FAGEN workshops: output fabrication as a labelled failure type. https://arxiv.org/abs/2607.04686
Model Context Protocol, Specification 2026-07-28: Tools (isError tool execution errors, outputSchema, client timeouts). https://modelcontextprotocol.io/specification/2026-07-28/server/tools
