Build an AI Agent, Stage 1: Define the Problem Before You Write a Prompt
An agent built against a one-line brief will be tested against the same line. If the brief names the wrong number, the agent can work perfectly and still fail, and nobody will know until the number doesn't move.
The example in this series is a composite scenario, built from patterns common to requirements intake in regulated organisations. It is not a single real organisation, and its figures are illustrative. The code is real: every output block below was produced by running it.
The brief from the head of change at a mid-sized insurer was one sentence and a target. Use an AI agent to triage incoming change requests and clear the backlog. Target: triage time down 50%. An engineer had a prototype by the end of the week. It read a request, labelled its type and suggested an owner in under a minute. A business analyst took about 25 minutes to do the same reading and classifying. On paper, the target was beaten several times over.
The senior business analyst asked for one more week before anything was built properly. She pulled twenty change requests that had been closed in the last quarter and traced each one backwards: when it arrived, when it became ready for the product owner to size, and what had happened in between. The median from submission to "ready to size" was six working days. About 25 minutes of that was an analyst reading and classifying. Most of the rest was waiting: eight of the twenty had needed a clarifying round with the requester before anyone could size them, and those eight had needed a median of two rounds each.
A classifier with zero latency would have removed 25 minutes from a six-day number. The target in the brief couldn't be reached by the thing being built, and the thing that would move it, asking for every missing fact once instead of in instalments, wasn't in the brief at all. The trace found something else the brief had no field for. Two of the requests that had gone straight to an owner later turned out, in build, to contradict a requirement the business had already accepted.
This is Stage 1 of Build an AI Agent From Scratch. Its job is to replace the one-line brief with a problem definition that a later stage can test against.
Why the one-line brief feels like enough
The brief wasn't careless. The head of change knew the pain precisely: a growing backlog and product owners complaining that requests reached them half-formed. "Triage time" sounded like a number anyone could measure, and "an AI agent" was the solution the organisation had budget and appetite for. Writing more would have felt like delaying the work.
What the brief did, without anyone intending it, was make three decisions early. It chose the design (an agent), before anyone had asked whether the work needed one. It chose the user as "the business", which is nobody in particular. And it chose a metric that measured the part of the process that was already fast.
A problem definition is six fields, and each one can be tested
Stage 1 produces six fields. None of them is new. What makes them useful is that each one is written so that a check can fail it:
Field | What it must say | How it fails a check |
Objective | The outcome, stated without mentioning AI, plus what must be preserved | Names a solution; preserves nothing |
User | The person who acts on the output, the decision they make and the window they make it in | "The business"; no decision; no time window |
Workflow | The current steps, as people actually do them | Empty, or copied from a procedure nobody follows |
Trigger | The observable event that starts the work | "Requests come in" |
Outcome | An end state you could inspect in a system | A comparison: "faster", "better" |
Success criteria | Each with a measure, a baseline, a numeric target and a data source; at least one says what must not happen | No measure, no baseline, all throughput |
The checks aren't a house invention. NIST's AI Risk Management Framework opens its MAP function with the same demand: "Intended purposes, potentially beneficial uses, context-specific laws, norms and expectations, and prospective settings in which the AI system will be deployed are understood and documented", including "the specific set or types of users along with their expectations" (MAP 1.1). It asks for "the business value or context of business use" to be "clearly defined" (MAP 1.4) and for the "targeted application scope" to be "specified and documented" (MAP 3.3) (NIST, 2023). What NIST leaves to the practitioner is how to tell whether the documentation is good enough. A checker is one answer.
The series keeps the problem definition as a JSON file next to the agent's code, and a small standard-library script, blueprint_check.py, runs on it the way a linter runs on source. Here is the sponsor's brief typed into the six fields exactly as given, then checked:
$ python blueprint_check.py problem.v0.json
problem.v0.json (problem definition, v0.1)
FAIL objective names a solution ('AI'), not an outcome
FAIL must_preserve names nothing the process must keep intact
FAIL user.primary 'the business' is not a person who acts on the output
FAIL user.acts_on_output no decision named that the output feeds
FAIL user.window no time window in which the user acts
FAIL trigger 'Requests come in.' names no observable event or channel
FAIL workflow no as-is steps; Stage 2 has nothing to classify
FAIL outcome 'Faster triage.' is a comparison, not an end state you could inspect
FAIL success_criteria.SC1 no baseline: nobody can say whether the target is an improvement
FAIL success_criteria.SC1 no data source: where would the number come from?
FAIL success_criteria.SC2 no measure: 'Improve requirement quality' cannot be tested
FAIL success_criteria.SC2 no baseline: nobody can say whether the target is an improvement
FAIL success_criteria.SC2 target 'better' has no number
FAIL success_criteria.SC2 no data source: where would the number come from?
FAIL success_criteria every criterion is throughput; none says what must not happen
FAIL out_of_scope empty: nothing the system refuses to do
FAIL evidence 0 closed cases traced; trace at least 20
FAIL evidence nobody interviewed does the job today
verdict: FAIL (18 fail, 0 warn)
Eighteen failures from two sentences says little about the sponsor and a lot about the decisions the brief left to whoever builds first. The checker can't tell whether a field is right. It can tell whether a field could ever be tested, which is the property every later stage depends on.
The test: put your own agent's problem statement into the six fields. Count the fields a check could fail today. Each one is a decision someone will make by default during the build.
The discovery interview that fills the fields
Filling the fields takes two pieces of discovery work. The first is an interview, about 75 minutes, with the people who do the job today. At the insurer that meant two business analysts, a product owner, and the compliance intake lead. The sponsor joined for the last fifteen minutes to hear the result, not to supply it.
The questions map onto the fields:
Field | The question that fills it | Who answers |
User | "When a triaged request reaches you, what do you decide, and how soon do you need to?" | Product owner |
Workflow | "Walk me through the last request you handled, step by step. Where did you stop and wait?" | Analysts |
Trigger | "Where do requests arrive, and what do the ones from each channel look like?" | Analysts |
Outcome | "What has to be true of a request before you can size it?" | Product owner |
Must preserve | "What do you check on every request that isn't written down anywhere?" | Analysts, compliance |
Guards | "What's the worst mistake you've seen in this process, and how was it caught?" | Everyone |
The last two questions come from Verification-First Agent Design, and they earned their place again here. The compliance intake lead said a regulatory request must never lose its deadline or its regulator reference, whatever else happens to it. That became the must_preserve list.
The senior analyst's answer to the worst-mistake question was more specific. A broker team had once asked to see claim status in the broker portal. In triage, an analyst summarised it as "brokers to view and update claim status". The summary was what the product owner sized and the team built, and the update capability went through two sprints before anyone compared it with the original request. Nobody had invented anything on purpose. A paraphrase had added a verb. That story became a guard criterion, and in Stage 3 it becomes the agent's top failure mode: an invented requirement.
The test: for your agent, ask the worst-mistake question of two people who do the job today. If the answer describes a failure your current success criteria wouldn't detect, you have found a missing guard.
The backwards trace, and the number it changed
The second piece of work is the trace, the move set out in Discovery for AI Features: Scoping Before You Prompt. Pull twenty closed cases. For each, ask what happened next and what decided it, then check whether the deciding fact was in the request as submitted.
What happened next | Requests | What decided it | In the request as submitted? |
Routed straight to an owner | 6 | Product line and system named on the form | Yes |
Clarifying round with the requester first | 8 | A missing effective date, product line or regulation | No: missing |
Linked to an existing backlog item | 3 | An open item already in the backlog | No: in the backlog |
Sent to compliance | 2 | A regulator reference or a near deadline | Yes |
Rejected as a support ticket | 1 | It asked to fix one policy record | Yes |
Eleven of the twenty turned on a fact that wasn't in the request: eight because something was missing, three because the answer lived in the backlog. That is the work that takes time, and none of it is classification. The two conflicts that surfaced in build add a third kind: facts that live in the accepted requirements, which nobody checked at intake.
So the success criteria changed. "Triage time down 50%" became a criterion about the whole wait (share of requests ready to size within two working days, baseline 6 of 20) and one about the rounds that cause it (median clarifying rounds, baseline 2, target 1). The conflicts became a third (conflicts flagged at intake, baseline 0 of 2). And the broker story and the compliance lead's rule became two guards: triage never adds scope or misquotes a requirement, and regulatory requests reach compliance within a working day.
The problem definition, after
Here is part of the rewritten file. The full version holds all eleven workflow steps the interview surfaced, including two the sponsor's brief had never mentioned: acknowledging the requester and merging duplicate records.
{
"objective": "Every incoming change request reaches its product owner ready to size: classified, linked to the existing requirements it touches, and with any missing facts asked for in one round, without adding, dropping or changing anything the requester asked for.",
"user": {
"primary": "product owner (one per product line)",
"acts_on_output": "decides whether to accept the request and size it",
"window": "within 2 working days of submission"
},
"trigger": "A change request is submitted through the web form or arrives in the change-request mailbox.",
"must_preserve": ["the requester's original wording, stored unedited", "deadline and regulatory-reference fields", "the priority set by the requesting business unit"],
"success_criteria": [
{"id": "SC1", "measure": "share of requests with status 'ready to size' within 2 working days",
"baseline": "6 of 20 traced requests", "baseline_n": 20, "target": ">= 70% over a rolling 4 weeks",
"source": "backlog tool status history, weekly query"},
{"id": "SC4", "guard": true,
"measure": "triage outputs asserting scope the request does not contain, or quoting a requirement inaccurately",
"baseline": "1 scope-changing paraphrase in 20 traced requests", "baseline_n": 20,
"target": "0 in a weekly sample of 20; each one is an incident",
"source": "weekly analyst sample plus the automated grounding check"}
]
}
The objective no longer mentions AI. It names what the product owner receives and, after "without", what the process must not do to the request on the way. And the checker's verdict:
$ python blueprint_check.py problem.v1.json
problem.v1.json (problem definition, v1.0)
WARN success_criteria.SC2 baseline rests on 8 case(s); re-baseline after launch
WARN success_criteria.SC3 baseline rests on 2 case(s); re-baseline after launch
WARN success_criteria.SC5 baseline rests on 2 case(s); re-baseline after launch
verdict: PASS (0 fail, 3 warn)
The warnings are honest and should stay. A baseline of "0 of 2 conflicts found at intake" is a real observation, but it is two cases. It is enough to justify the criterion and not enough to judge a target against. The fix is to re-baseline from the first months of live data, and the warning keeps that from being forgotten.
How to evaluate your own problem statement
Run these against the agent closest to your production. Each has an answer you can check:
Does the objective still make sense if a person did the work? If it mentions a model or an agent, rewrite it.
Can you name the person who acts on the output, the decision, and how soon they make it?
Was the workflow written by watching or tracing real cases, or copied from a procedure?
Does every success criterion have a baseline from real cases, and a query someone owns that will produce the number?
Does at least one criterion say what must not happen, with a target of zero or close to it?
Have you traced twenty closed cases, and does the elapsed time sit where your headline metric assumes it does?
Question 6 is the one that changed the insurer's project. It costs a day.
When Stage 1 is overkill, and where it bites back
Approach | Gains | Costs | Use when |
One-line brief | Fast to a prototype | The metric may measure the wrong part of the process; guards are left out | A throwaway spike to see what a model can do |
Six fields from the sponsor alone | Better than one line | Users, workflow and guards come from the person furthest from the work | Never on its own |
Six fields plus interview and trace | Criteria you can test; guards from people who know the failures | One to two days before build | Anything that will touch a system of record, a regulated process or another team's queue |
Exceptions. A time-boxed capability spike doesn't need Stage 1, as long as everyone agrees the code will be thrown away. A tool whose only user is its builder can skip the interview, though not the trace. And where a written rule already governs the work and is demonstrably followed, the trace mostly re-reads the rule. Check that the rule exists before skipping it.
Where it bites back. A problem definition can be gamed like any document. Baselines get chosen to make targets easy, and guard criteria get set to zero with no one sampling for them. The source field exists to stop the second: if nobody owns the query, the criterion isn't real.
How Stage 1 goes wrong
The sponsor writes it alone. The people furthest from the work supply the user, the workflow and the guards. The checker's evidence rule (someone interviewed must do the job today) exists for this.
Every criterion is throughput. Speed and volume targets with nothing about what must not happen. The agent will be optimised for the only numbers anyone looks at.
Baselines from memory. "About a week" is a guess. Twenty traced cases is a number, and it can be wrong in ways you can find.
The workflow from the procedure manual. The trace found the merge step and the acknowledgement step because it followed real requests. The manual had neither.
What to do next
Take the agent closest to production. Write its problem as the six fields, then trace twenty closed cases with the two questions and a last column reading in the input: yes or no. If you want a starting point, the checker's rules are listed in the table above; they take about an hour to encode for your own format. Then move on to Stage 2: Decide Whether an Agent Is Appropriate, which takes the workflow you just wrote and decides, step by step, what each part should be.
Final takeaway
A one-line brief chooses the design, the user and the metric before anyone has looked at the work. Six fields, an interview and twenty traced cases replace those choices with evidence. The definition that comes out may point somewhere the sponsor didn't expect, which is the reason to write it before the prompt.
Sources
NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023. MAP 1.1 (intended purposes, users and deployment context documented), MAP 1.4 (business value or context of use defined), MAP 3.3 (targeted application scope specified and documented).
Series navigation. Previous: the series hub. Next: Stage 2: Decide Whether an Agent Is Appropriate. The full stage list is on the Build an AI Agent From Scratch hub.
Frequently Asked Questions
What should a problem statement for an AI agent contain?
Six things: the objective stated without mentioning AI, the person who acts on the output and the window they act in, the current workflow step by step, the event that triggers the work, the end state you could inspect, and success criteria that each have a measure, a baseline, a numeric target and a data source. At least one criterion should say what must not happen.
How do you find the real problem behind an agent request?
Trace twenty recently closed cases backwards. For each, ask what happened next and what decided it, then check whether the deciding fact was in the input the agent would receive. Combine that with a short interview of the people who do the work today, asking what they check that isn't written down and what the worst mistake they have seen was.
Why shouldn't the objective mention AI?
An objective that names the solution decides the design before anyone has checked whether an agent is the right design. Stated as an outcome, the same objective lets Stage 2 conclude that most of the work is rules, which is often the cheaper and more testable answer.
Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.
