Build an AI Agent From Scratch: Fifteen Stages, One Agent, a Test for Each
Updated: 22 minutes ago
Most agent tutorials start at the loop: a model, a prompt, some tools. The decisions that later cause the expensive failures are made before any of that, so this series starts three stages earlier and closes every stage with a test.
The agent built in this series is a composite example, built from patterns common to requirements intake in regulated organisations. It is not a single real organisation, and its figures are illustrative. The code is real: every output shown in the series was produced by running it.
The requirements-intake agent at a mid-sized insurer has appeared on this site twice, and it failed both times. In Verification-First Agent Design, it merged a regulator-driven change request into a year-old duplicate, and the merge dropped the deadline. In How to Test an AI Agent, it arrived two weeks before release with a single, illustrative 92% evaluation score that nobody could connect to anything it had done in the backlog.
Neither failure was a coding mistake. The merge happened because nobody had written down that a deadline must survive triage, so merging was granted as a convenience. The meaningless score happened because nobody had written down what the agent must never do, so no test case asked. Both were decisions that belonged to stages the team skipped on the way to a working demo.
This series builds the same agent again, from a blank page, in an order where those decisions come first. It runs to fifteen stages. The first three are published here, with the agent's complete blueprint.
Why tutorials start at the loop
There are good reasons to start with code. A loop that calls a tool and reads the result is the most satisfying thing to see working, and it answers the first question everyone asks: can a model do this at all? Our own Building AI Agents tutorials start there, with What Is an Agent (With Code) and then Build a Single-Tool Agent. Those tutorials are good at what they teach, and later stages of this series use them.
The trouble is what that order leaves to chance. When the first artefact is a working loop, the problem statement is whatever the demo happened to solve, the tool grants are whatever made the demo work, and the test set is drawn from the demo's cases. Each of those is a decision, and none of them was made on purpose.
Each stage produces the input to the next stage's test
The series works because each stage leaves behind an artefact, and the next stage's test reads it. Stage 1 writes the problem definition, including success criteria with measures and baselines. Stage 2 reads that problem's workflow and decides, step by step, which parts need an agent at all. Stage 3 takes the steps that do and writes the agent's responsibilities, permissions and hand-over cases, which become the Requirements Agent blueprint. Stage 4's instructions are tested against the blueprint's prohibited actions. Stage 5's tools are tested against its permissions. By Stage 11, most of the test suite has already been specified by the stages before it.
That is the verification-first rule applied to a whole build: no stage is finished until it names its test and the test passes. A stage that produces a document nobody checks is not finished, and the series treats it that way. The design documents are kept as JSON next to the code, and a checker runs on them in CI the way a linter runs on source.
The running example
The agent triages incoming change requests for the business-analysis team at a mid-sized insurer. Business units raise several hundred requests a quarter through a web form and a shared mailbox. Each one has to be classified, checked against the existing requirements for duplicates, overlaps and conflicts, made complete enough to size, and routed to the product owner for that line.
It is a good teaching case because it is ordinary work with real risk and no customer in the loop. Two failures matter most. Misrouting sends a request to the wrong owner, and it sits there. An invented requirement is worse: a paraphrase or a leading question that adds scope the requester never asked for, which then gets built. The target is Level 2 on the autonomy ladder from What Is AI Agent Management?: the agent prepares, and a person approves before anything takes effect.
By the end of Stage 3, the "agent" is much smaller than the original brief. Most of the workflow turns out to be rules and two bounded model steps, and the agent does one thing: it investigates how a request relates to existing requirements and drafts the questions that would make it ready to size.
The fifteen stages
The stages fall into four bands. The first decides what to build and whether to build an agent. The second adds capabilities one at a time. The third puts controls and evidence around them. The fourth runs the agent and keeps it honest.
Stage | What it produces | The test that closes it | Status |
1. Define the problem | Objective, user, workflow, trigger, outcome, success criteria | Every success criterion has a measure, a baseline and a data source; 20 closed cases traced | Available |
2. Agent or not | A design for every workflow step: automation, model step, person or agent | Every step's answer to the five questions quotes its evidence | Available |
3. Responsibilities | The Agent Design Canvas and the Requirements Agent blueprint | The blueprint checker passes with every field filled | Available |
4. Instructions | A production Prompt Spec | The spec linter passes; the test pack has a case of every kind | Available |
5. Tools | Tool contracts behind a gateway | Every prohibited call is refused at the tool layer | Available |
6. Skills | Packaged, separately tested capabilities | Each skill passes its own test pack | Available |
7. Memory | A decision on what persists, and for how long | A stale memory never overrides the current request | Available |
8. Retrieval | Search over the requirements repository | No match returns "none", never an invented requirement | Available |
9. Workflow | Agent, tool, verification, decision, action | Fail a step; the run stops or hands over | Available |
10. Human approval | Gates with one question each | Planted bad drafts are caught at the gate | Available |
11. Testing | The full suite, including boundary and adversarial layers | Must-never cases pass on 5 of 5 runs | Available |
12. Evaluation | Golden set, calibrated judge, promotion bar | The judge is calibrated before its score is read | Coming |
13. Observability | A run record that answers four readers | Each reader's question answered from one trace | Coming |
14. Deploy | Versioned release, staged rollout, rollback | Rollback rehearsed before the canary starts | Coming |
15. Monitor and improve | Drift signals and demotion triggers | Seeded drift trips the alert and the demotion | Coming |
What to read while the later stages are written
Every later stage builds on a curriculum article that is already live. If you want to work ahead, these are the ones each stage will apply to the Requirements Agent:
Stage 4, instructions: Stage 4: Write the Instructions the Blueprint Can Test, building on The Prompt Spec and Prompt Regression Testing in CI.
Stage 5, tools: Stage 5: Give It Tools, and Prove What Each One Does, building on Designing Tool Contracts and Least Privilege for AI Agents and the hands-on Add Multiple Tools to an Agent.
Stage 6, skills: Stage 6: Package Its Skills So Each One Is Tested Alone, building on Evaluating AI Agents for sizing each skill's test pack.
Stage 7, memory: Stage 7: Decide What It Remembers, and Test What It Forgets, building on Agent Memory and State and the budgeting rules in Context Engineering: A Practitioner's Guide.
Stage 8, retrieval: Stage 8: Retrieval That Can Say "None", building on Add Embeddings and Search and Build a Basic RAG System.
Stage 9, workflow: Stage 9: A Workflow Where Every Failure Stops or Hands Over, building on Build a Planning Agent and Handle Agent Failures.
Stage 10, human approval: Stage 10: Human Approval Gates That Catch Planted Mistakes, building on Where Human Approval Belongs: the Autonomy Ladder Per Action.
Stage 11, testing: Stage 11: A Test Suite Where Must-Never Means Never, building on How to Test an AI Agent and Agent Guardrails and Safety.
Stage 12, evaluation: Evaluating AI Agents and Evaluate Agent Reliability.
Stage 13, observability: The AI Agent Observability Checklist and Production Observability for Agents.
Stage 15, monitoring: Monitor for Prompt Drift and Continuous Evaluation in Production.
Stages 6 and 14 (skills and deployment) are new material with no live article yet.
The code grows with the series
Stages 1 to 3 ship a small folder of standard-library Python. The problem definition, the workflow decisions and the blueprint are JSON files. Three scripts check them: blueprint_check.py flags missing and untestable fields, agent_or_not.py applies the decision tree to each workflow step, and grounding_check.py refuses any triage note whose quotes don't match their sources. Twenty-one unit tests prove each rule fires when its field is broken. Later stages add rules to the same checker and tests to the same suite, so by Stage 11 the suite is a regression gate for the agent, not a separate project.
The test for this page: take an agent you are building or running, and list which of the fifteen stages produced a written artefact. Any stage without one was decided by default. That gap is where to start.
How to use the series, depending on your job
If you analyse requirements or run delivery, Stages 1 and 2 are your work, done with sharper tools. The backwards trace and the discovery interview in Stage 1 are requirements elicitation pointed at a process an agent will touch.
If you test or assure software, read Stage 3 closely. Every field in the blueprint is written so a test can read it, and the checker is the first test in the suite.
If you build agents, resist skipping to Stage 4. The stages before it decide which tools you are allowed to write.
What to do next
Start with Stage 1: Define the Problem. Before you read it, write down the problem statement for the agent closest to your own production, as you would put it to a sponsor. Stage 1 runs that statement through a checker. Keep your version, because you will want to compare.
Final takeaway
An agent is the sum of decisions about what it may do, what it must never do and how anyone would know. Most builds make those decisions by default, in the order the code happens to need them. This series makes them on purpose, in an order where each one becomes the test for the next.
Series navigation. This is the series hub. Next: Stage 1: Define the Problem. Then Stage 2: Decide Whether an Agent Is Appropriate and Stage 3: Define Responsibilities, which ends in the full Requirements Agent blueprint.
Now available: Stage 4: Write the Instructions the Blueprint Can Test, which traces every rule to a test; Stage 5: Give It Tools, and Prove What Each One Does, which proves what each tool really changes; Stage 6: Package Its Skills So Each One Is Tested Alone, which gates each capability on its own bars; Stage 7: Decide What It Remembers, and Test What It Forgets, which makes stale memories impossible to recall; Stage 8: Retrieval That Can Say "None", which tests search on requests with nothing to find; Stage 9: A Workflow Where Every Failure Stops or Hands Over, which injects a fault at every step; Stage 10: Human Approval Gates That Catch Planted Mistakes, which measures each gate with known-bad drafts; and Stage 11: A Test Suite Where Must-Never Means Never, which judges must-never rules apart from the average.
Frequently Asked Questions
What are the stages of building an AI agent?
This series uses fifteen, in four bands. Define: the problem, whether an agent is appropriate at all, and its responsibilities. Build: instructions, tools, skills, memory, retrieval and workflow. Control and prove: human approval, testing and evaluation. Run: observability, deployment and monitoring. Each stage ends with a test that has to pass before the next one starts.
Why does the series start before any code is written?
Because the agent failures that cost the most come from decisions made before the code: what the agent must never do, which actions it may take on its own, and who it hands over to. If those decisions are not written down first, the build makes them by default and the tests inherit the gaps.
Do I need to read the stages in order?
The stages are written to be read in order, because each one uses the artefact the previous one produced. If you already have an agent in production, start with Stage 3 and run the blueprint checker against what you have; the gaps it reports point you back to the stage that fills them.
Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.
