AI Agents: What They Are and Why They Keep Failing
Updated: 5 days ago
An AI agent is a loop that takes actions in other systems. The parts that decide whether it is safe to leave alone (budgets, permissions, state, stop conditions, traces) sit outside the model, and most agents ship without them.
The example in this article is a composite scenario, built from patterns common to finance operations and continuing the distributor's agents used elsewhere in this curriculum. It is not a single real organisation, and its figures are illustrative. The code is real: the output below was produced by running it.
In the second week of its pilot, the duplicate-invoice investigator at a mid-sized distributor got a routine case. An invoice for 7,960.00 from one supplier was booked in the UK entity. A posting for the same amount, from the same supplier, had turned up in the Irish entity. Was it the same bill keyed twice?
The agent searched both entities and found both postings. Then it called the supplier-statement archive, which is how the clerks decide these cases: if the supplier billed once, the second posting is a duplicate. The archive was being reindexed that night and every call timed out. The agent tried again with a different date range, then another, five times in all. Then it wrote its finding: "Two separate invoices; consistent with the supplier statement."
It had never seen the statement. The clerk read a confident, well-formed note, closed the case, and the Irish posting went out in the next payment run. Month-end reconciliation caught it and the supplier refunded the money, but working out what had happened took two days. The run log held only the final note. The five timeouts were found in the archive's own access log.
Nobody had built anything careless. The agent was what most agent tutorials describe: a model, a prompt, three tools and a loop. That description is accurate, and it is the reason the pilot failed.
What an AI agent is
The textbook definition is broad. Russell and Norvig's Artificial Intelligence: A Modern Approach defines an agent as "anything that can be viewed as perceiving its environment through sensors and acting upon that environment through effectors" (Russell and Norvig, 1995). A thermostat qualifies. Builders need something narrower.
Anthropic's widely cited guide separates two designs. Workflows are "systems where LLMs and tools are orchestrated through predefined code paths". Agents are systems "where LLMs dynamically direct their own processes and tool usage", and in practice are "typically just LLMs using tools based on environmental feedback in a loop" (Anthropic, 2024). This curriculum uses a working definition built on that one:
An AI agent is a system that uses a model to choose actions toward a goal, carries them out through tools, observes the results, and continues until it stops or hands over.
As a loop: goal, decide, act, observe, update state, then stop, hand over or go round again. Only decide is the model. The rest is software, and that is where production agents usually break. If the steps are known in advance, you probably want a workflow. Agent, Workflow, Automation or RPA? gives a five-question decision tree for that choice, and it is how the distributor's team ended up with a read-only investigator in the first place.
Why "a model, tools and a loop" feels like enough
The three-part description is correct about the mechanism, and you can build one in an afternoon. The loop in What Is an Agent (With Code) fits in a few dozen lines, and watching it work is genuinely useful: the model reads a result, changes its plan and tries something else, which no script does.
Frameworks reinforce the picture: declare the tools, write the instructions, and the framework runs the loop. Demos reinforce it more, because they run on inputs chosen to work, against tools that respond.
What it leaves out is everything that decides what happens when a step goes wrong. In the investigator's loop, the only way a run could end was the model deciding it was finished. With the evidence unavailable, the model produced the most plausible ending, and the most plausible ending was a verdict.
Why agents keep failing: the loop multiplies small errors
Every step is a chance to be wrong, and later steps build on earlier ones. As an illustrative calculation, if each step succeeds 95% of the time and the steps fail independently, a ten-step run finishes cleanly 0.95¹⁰ ≈ 60% of the time. Real errors are not independent: one bad observation tends to poison the steps after it.
The measured picture is consistent with that. τ-bench, a benchmark of agents serving simulated customers with tools and written policies, introduced a metric called pass^k: a task counts only if the agent succeeds on all k attempts. Its authors reported in June 2024 that "even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)" (Yao et al., 2024). Models have improved since, so treat the figures as dated. Anthropic's own guidance names "the potential for compounding errors" as a cost of agent autonomy and recommends "extensive testing in sandboxed environments, along with the appropriate guardrails" (Anthropic, 2024).
The earlier version of this article listed the usual failures: context loss, hallucinated actions, poor error recovery, compounding errors and overconfidence. The investigator's run shows them arriving together. Recovery meant retrying the same failing call with small variations. Timeout messages filled the context where evidence should have been. The model summarised a statement it had never retrieved, in the same confident register it uses for verified facts, and the run ended looking complete when it was partial.
None of those would have been fixed by better wording in the prompt. Each one traces to a component the system did not have: a retry limit, a record of which checks had been done, a rule that a verdict needs evidence, a way to end the run other than a verdict, and a trace.
The test: list every way a run of your agent can end. If the only exit is the model saying it is done, the model is also your stop condition.
The anatomy of a production agent
A production agent is the loop plus the parts that make each step safe to leave unattended. The table maps ten components to the failure each one prevents and the check that shows it is present. The right-hand column names the part of this curriculum that covers it in depth.
Component | What it is | Failure it prevents | Check you can run | Covered in depth by |
Goal and loop | The runtime that runs decide, act, observe, update | A run with no defined end | Every exit is a named state in code | This article |
Model | A pinned model version | Behaviour changes with no release | The version is recorded on every run | Prompt Regression Testing in CI |
Instructions | A Prompt Spec: purpose, rules, examples, hand-over cases | Rules that exist only in people's heads | The spec has a hand-over section and tests | The Prompt Spec |
Context | What is assembled into the window for each decision | Stale, conflicting or excess text outranks the goal | You can print the exact context for any step | Context Engineering: A Practitioner's Guide |
Tools and credentials | Contracts per tool, an identity per agent | Excessive functionality and permissions | Deleting the prompt's "never" line changes nothing | Tool Contracts and Least Privilege |
Autonomy per action | A ladder level for each action, with approval gates | Actions running at a level nobody approved | Each action's level is written down with its evidence | What Is AI Agent Management?; Where Human Approval Belongs |
State | The runtime's record of what is known and missing | Repeating work, forgetting a constraint, claiming unchecked facts | The runtime, not the model, can say what is still missing | Agent Memory and State |
Stop conditions and escalation | Step and retry budgets, evidence rules, a hand-over outcome | Retry loops, runaway cost, confident guesses | A forced tool failure ends in a hand-over | Verification-First Agent Design |
Observability | One trace record per step | Incidents nobody can reconstruct | Rebuild any run from last week from the trace alone | Production Observability for Agents |
Evaluation and tests | Outcome tests run several times; evaluation against labelled cases | Shipping on a demo and a single score | Each "must never" has a test that checks the system | How to Test an AI Agent; Evaluating AI Agents |
The ten are easier to hold as four groups, because each group answers one question someone will ask after an incident.
What did it know when it decided? Model, instructions, context
The model should be a pinned version, recorded on every run, because a provider update can change behaviour with no change on your side. Prompt Regression Testing in CI treats a model upgrade as a change that goes through the same gate as a prompt edit.
The instructions should be written as a specification rather than a paragraph. The Prompt Spec sets out nine parts, and the one that mattered here is uncertainty and escalation: what the agent does when it cannot finish. The investigator's prompt said what a duplicate was. It said nothing about what to do when the evidence was unavailable.
Context is everything assembled into the model's window at each step, and it is finite in a way that matters. Anthropic's applied AI team describes research showing that "as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases", and argues that context "must be treated as a finite resource with diminishing marginal returns" (Anthropic, 2025). Five timeout messages are five chunks of noise where the evidence should be. Context Engineering: A Practitioner's Guide shows how to budget it, pin the rules that must survive and log what the model saw.
The test: for any step in last week's runs, can you print the exact model version, instructions and context the model saw? If not, you can't explain any decision it made.
What could it change? Tools, credentials, autonomy
Tools are where a model's choice becomes a change in someone else's system, so they are where the limits belong. OWASP names three root causes of excessive agency in LLM applications: excessive functionality, excessive permissions and excessive autonomy. Its mitigations include restricting tools to what the job needs, granting minimum permissions and applying authorisation in downstream systems rather than relying on the model to decide (OWASP, 2025). Designing Tool Contracts and Least Privilege for AI Agents turns that into a seven-clause contract for each tool, enforced in a gateway, using this same investigator after its promotion.
Autonomy is set per action, not per agent. On the ladder in What Is AI Agent Management?, Level 1, Assist, means the agent "finds, summarises, recommends" and people decide and act. Level 2, Draft, means it "prepares the action" and people "approve each one before it executes". Level 3, Execute, means it "performs approved low-risk actions after deterministic checks". The investigator ran at Level 1, so a clerk made every final call, which is meant to be the safeguard. In practice the clerk had no way to see that the conclusion rested on nothing, which is why approval gates need the trace beside them. Where Human Approval Belongs shows how to score each action and keep the remaining approvals from becoming rubber stamps.
The test: for each tool, pull the credential's real permission report and compare it with the actions the agent's job needs. Anything extra is risk you carry for the demo's convenience.
When does it stop? State, stop conditions, escalation
State is the runtime's record of the task: which checks have been done, what they returned, what is still missing. This is the group the pilot lacked entirely. State has to live in code, not only in the model's context, because the model's account of what it had checked is exactly what failed. Agent Memory and State covers the mechanics.
Stop conditions are rules in code that end a run: a step budget, a retry budget per tool, and a rule that a verdict needs its evidence in state. Anthropic notes that "it's also common to include stopping conditions (such as a maximum number of iterations) to maintain control", and that agents "can then pause for human feedback at checkpoints or when encountering blockers" (Anthropic, 2024). An iteration cap on its own would not have saved the investigator. It stopped after seven calls, well inside any default cap, and still produced a wrong answer, because the cap limits how long a run lasts, not what it is allowed to conclude.
Escalation is the third outcome. A case can be a duplicate, not a duplicate, or not decided yet, and here is why. The clerks already had that outcome, and the discovery question that finds it is short: "What do you write on a case you couldn't finish?" Their answer was "statement pending". It was a status in their tracker, and nobody had put it in the agent's list of outcomes. Verification-First Agent Design calls these the "should hand to a person" behaviours and asks for them before the design exists.
The test: make one of your agent's tools fail on purpose in a sandbox. If the run ends in anything other than a hand-over that names what was not checked, you have found your first fix.
How would you know? Traces, evaluation, tests
A trace is one record per step: what the model was given, which tool it called with which arguments, what came back and what the runtime decided. With one, the investigator's incident takes minutes to read. Without one, it took two days. Production Observability for Agents covers what to capture.
Tests check the system rather than the agent's account of it, and run each case several times, because one pass is a sample, not a result. How to Test an AI Agent sets out the verification chain and pass^k in practice. Evaluating AI Agents covers the labelled sets and calibrated judges that decide whether an agent is good enough to promote.
The test: pick a run from last week and rebuild it from the trace alone. If you need the model, the vendor or the person who built it, you don't have a trace.
The investigator, rebuilt
The rebuild kept the same model behaviour and tools and changed the runtime around them, sketched below in standard-library Python. The model is a labelled stand-in: a deterministic policy that reproduces the pilot's behaviour (retry the failing tool with a new date range, then answer anyway), so the output can be reproduced exactly. The systems are stand-ins too. The point is the loop.
def anatomy_loop(world, goal):
state = {"evidence": {}, "failures": {}, "status": "running"} # memory the runtime owns
trace, observations = [], []
def log(step, event, **detail): # one record per step
trace.append({"step": step, "event": event, **detail})
for step in range(1, STEP_BUDGET + 1): # step budget
name, args = stand_in_model(goal, observations)
if name not in POLICY: # tool gateway: deny by default
log(step, "refused", tool=name, reason="no such tool")
continue
if name == "submit_finding": # stop condition checks STATE,
missing = sorted(REQUIRED_EVIDENCE - state["evidence"].keys()) # not the model's claim
if missing:
log(step, "refused", tool=name, outcome=args["outcome"], reason=f"missing evidence: {missing}")
return hand_over(state, trace, step, f"verdict claimed without {missing}")
log(step, "finished", outcome=args["outcome"])
return {"outcome": args["outcome"], "note": args["note"], "trace": trace}
k = key(name, args)
if state["failures"].get(k, 0) >= POLICY[name]["max_failures"]: # retry budget per tool
log(step, "refused", tool=k, reason="retry budget spent")
return hand_over(state, trace, step, f"{k} unavailable after {state['failures'][k]} attempts")
try:
result = call_tool(world, name, args)
state["evidence"][k] = result
observations.append({"tool": k, "ok": True, "result": result})
log(step, "ok", tool=k, args=args, rows=len(result))
except Exception as e:
state["failures"][k] = state["failures"].get(k, 0) + 1
observations.append({"tool": k, "ok": False, "result": str(e)})
log(step, "error", tool=k, args=args, error=type(e).__name__)
return hand_over(state, trace, STEP_BUDGET, "step budget spent")
POLICY sets each tool's side-effect class and retry budget (two failures for the read tools), REQUIRED_EVIDENCE lists the three checks a verdict needs, and STEP_BUDGET is eight. Running the naive loop and the rebuilt loop against the same stand-in systems, first with the archive down and then with it up, prints this:
=== naive loop, statement archive down
{'outcome': 'not_duplicate', 'note': 'Two separate invoices; consistent with the supplier statement.', 'calls': 7}
=== anatomy loop, statement archive down
{'outcome': 'statement_pending', 'note': 'Handed to AP clerk: get_statement unavailable after 2 attempts.'}
{"step": 1, "event": "ok", "tool": "search_invoices:UK01", "args": {"entity": "UK01", "supplier": "S-1182", "amount": "7960.00"}, "rows": 1}
{"step": 2, "event": "ok", "tool": "search_invoices:IE02", "args": {"entity": "IE02", "supplier": "S-1182", "amount": "7960.00"}, "rows": 1}
{"step": 3, "event": "error", "tool": "get_statement", "args": {"supplier": "S-1182", "period": "2026-06"}, "error": "Timeout"}
{"step": 4, "event": "error", "tool": "get_statement", "args": {"supplier": "S-1182", "period": "2026-05..2026-06"}, "error": "Timeout"}
{"step": 5, "event": "refused", "tool": "get_statement", "reason": "retry budget spent"}
{"step": 5, "event": "handed_over", "reason": "get_statement unavailable after 2 attempts", "checked": ["search_invoices:IE02", "search_invoices:UK01"], "not_checked": ["get_statement"]}
=== anatomy loop, statement archive up
{'outcome': 'duplicate', 'note': 'Supplier statement lists 1 charge(s).'}
The naive loop reproduces the incident: seven calls, five of them failures, and a fluent wrong verdict. The rebuilt loop stops calling the archive after two failures and hands the case to a clerk as statement pending, listing what it did and did not check. When the archive is up, the same model reaches the right verdict, so the runtime is not making the agent timid. It is making it honest about what it knows.
The retry budget is not the only defence. One of the five tests that ship with the code raises the archive's retry budget to ten, as someone might to "fix" too many hand-overs. The model then runs out of ideas and claims a verdict, and the evidence rule refuses it and hands over anyway. As a mutation check, disabling that one rule makes exactly that test fail. A check that no test can break is not doing anything.
How to dissect your own agent
Take the agent closest to production and sit with its builder and someone who does the work today. For each row of the anatomy table, write down where the component lives and how you would show it works. Six questions find most of the gaps:
How can a run end? List every exit in code. A model-declared "done" should be one exit among several.
What happens when each tool fails? Force one failure per tool in a sandbox and read what the agent concludes.
Who keeps the record of what has been checked? If only the model's context does, the agent can claim checks it never ran.
What is the hand-over outcome called? Ask the people who do the job what they write when they can't finish. That label belongs in the agent's outcomes.
What can its credentials do? Compare the permission report with the actions in its job.
Can you rebuild last Tuesday's runs? From the trace alone, without asking anyone.
Count the questions with no answer. If the answers to questions 1 and 3 are both "the model", the agent is running on its own word.
Tradeoffs and exceptions
Design | Gains | Costs | Best suited for | Warning signs |
Model, tools and a loop (framework defaults) | Fastest to a working demo; flexible | Exits, state and limits are implicit; failures are fluent | Spikes to learn what a model can do | The spike is being hardened instead of rebuilt |
Full runtime around the model | Every failure has a named exit; incidents are readable | More code; more hand-overs at first; a hand-over queue someone must own | Anything that writes to a system of record, contacts customers or feeds a financial decision | Hand-over rate climbs and nobody reads the queue |
Workflow with one model step | Easiest to verify; code fixes the path | Can't handle open-ended investigation | Work whose steps are known in advance | Cases keep falling outside the fixed path |
Exceptions worth naming. A personal assistant that reads documents its user can already see, and only advises, needs little of this. Its user is the check, and its worst failure is a bad suggestion. A coding agent working in a disposable sandbox can rely on the sandbox as much of its boundary, as long as anything leaving the sandbox (a commit, a deploy, a message) passes through a contracted tool.
Where the anatomy strains. Budgets set too tight turn the agent into a hand-over machine, and a queue of "statement pending" cases nobody works is the old failure in a slower form. Track the hand-over rate per reason against a range set in advance, and treat a rise as a signal about a tool or data source. A trace nobody reads is storage, not observability: name the person who reads it weekly.
What to do next
Run question 2 this week. Pick your agent's most important tool, make it fail in a sandbox, and read the run's last output. If it says anything that sounds like a conclusion, add a per-tool retry budget and an evidence rule like the ones above, give the agent a hand-over outcome named after the one your team already uses, and write a test that forces the failure and asserts on the hand-over. For the wider readiness picture across evaluation, guardrails, observability and governance, the AI Readiness Assessment asks twenty-four questions, and AI Agents Fail in Production in the Same Five Ways shows the other recurring patterns.
Final takeaway
An agent is a model making choices inside a loop that you wrote. Most production failures happen in the parts of that loop that aren't the model, such as the missing retry limit, the missing record of what was checked and the missing way to say "not decided". Build those parts deliberately and give each one a test. Then a dead tool ends in a hand-over instead of a payment.
Sources
Anthropic (Schluntz, E. and Zhang, B.), Building effective agents, 19 December 2024. Definitions of workflows and agents; agents as "LLMs using tools based on environmental feedback in a loop"; stopping conditions and pausing for human feedback; compounding errors and sandboxed testing.
Anthropic Applied AI team (Rajasekaran, P., Dixon, E., Ryan, C. and Hadfield, J.), Effective context engineering for AI agents, 29 September 2025. Context rot; context as a finite resource with diminishing marginal returns.
Yao, S., Shinn, N., Razavi, P. and Narasimhan, K., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv:2406.12045, June 2024. The pass^k metric; reported success and consistency figures.
OWASP GenAI Security Project, LLM06:2025 Excessive Agency, 2025. Excessive functionality, permissions and autonomy; mitigations.
Russell, S. and Norvig, P., Artificial Intelligence: A Modern Approach, Prentice Hall, 1995, chapter 2, "Intelligent Agents". Later editions say "actuators" for "effectors".
Frequently Asked Questions
What is an AI agent?
An AI agent is a system that uses a language model to choose actions toward a goal, carries them out through tools, observes the results and continues until it stops or hands the work to a person. The model makes the choices; everything else in the loop, including the tools, permissions, state, stop conditions and logging, is software someone has to design and test.
What is the difference between an AI agent, a chatbot and a workflow?
A chatbot answers once and stops. A workflow uses a model inside a path that code fixes in advance, for example to classify or extract. An agent lets the model decide which step and which tool comes next. That freedom is what makes agents useful for open-ended work, and it is also what makes them harder to test, so a workflow is the better choice whenever the steps are known.
Why do AI agents keep failing in production?
Because a loop multiplies small errors, and most agents are shipped as a prompt, some tools and a while loop. As an illustrative calculation, ten independent steps that each succeed 95% of the time all succeed only about 60% of the time. Production agents fail less when the runtime around the model owns the step budget, the permissions, the task state and the decision to stop or hand over, and records a trace of every step.
Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.
