Build an AI Agent, Stage 4: Write the Instructions the Blueprint Can Test
An agent's instructions should be derived from its blueprint, not written from memory. Every input the blueprint marks untrusted, every action it prohibits and every case where it must hand over has to appear in the instructions, and each one needs a test case, before the prompt ships.
The example in this series is a composite scenario, built from patterns common to requirements intake in regulated organisations. It is not a single real organisation, and its figures are illustrative. The code is real: every output block below was produced by running it.
The first sandbox run went well for most of the morning. The Requirements Agent read incoming change requests, searched the requirements repository and the backlog, and drafted triage notes for the analysts to review. Then the repository search timed out on one request, a broker team's ask to add a renewal reminder 30 days before expiry. The agent's note read: "No related requirements found. Suggested questions: …"
The analyst reviewing it nearly sent the request on as new work. She checked only because she remembered a renewal-reminder change from the spring. There it was, accepted and half built.
The blueprint from Stage 3 had a rule for exactly this. Its third hand-over case says that when a search or read fails, times out or returns something unreadable, the agent hands over to the senior analyst on duty, with what was searched and the error, and it "never" reports "no related requirements". The prompt didn't. It had been written in an afternoon from the team's memory of what the agent was for, in the Prompt Spec format, and it passed the Prompt Spec linter. It just wasn't the blueprint.
This is Stage 4 of Build an AI Agent From Scratch. Stage 3 left a blueprint: goals, inputs with trust levels, seven prohibitions with their enforcement points, six hand-over cases with owners. This stage turns it into the agent's instructions, and closes when a checker can trace every one of those obligations to the prompt and to a test.
Why writing the prompt from memory feels like enough
By Stage 4 the team knows the agent well. They have argued about its scope for three stages, and the instructions feel like a write-up of decisions already made. The Prompt Spec gives the write-up a structure, nine sections from purpose to verification, and its linter confirms each section is present, an input is marked untrusted, an example shows a hand-over and the test pack has a case of every kind.
The first draft did all of that. It named the agent's purpose, marked the request text untrusted, listed most of the must-nots in a sentence, gave a normal, a borderline and an escalation example, and linked a four-case test pack. The linter said it passed.
What the linter can't know is what this agent's blueprint requires. It checks the shape of a prompt, and the shape was right. The obligations were somewhere else.
The test: take your agent's current instructions and its design document side by side. Count the hand-over cases in the design, then count the hand-over rules in the prompt. If the numbers differ, the prompt was written from memory.
A well-formed spec can still miss the blueprint
The gap opens for a simple reason: two documents describe the same agent, written at different times by different people, and nothing connects them. The blueprint is written in a design meeting with the business analyst lead, compliance and the product owner in the room. The prompt is written later, usually by an engineer, from what they remember of that meeting. Every obligation the meeting added and nobody repeated is a candidate for falling out.
Hand-over cases fall out first, because they describe the agent's edges, and people describe what an agent does, not where it stops. Trust levels on internal sources fall out next: the request text is obviously untrusted, but requirement text and backlog items are written by colleagues, so they feel safe, and the blueprint marked them untrusted for a reason the prompt's author wasn't there to hear. A requirement or backlog item can carry text that reads like an instruction, and the model reads it in the same stream as everything else. OWASP's guidance for indirect injection is to "separate and clearly denote untrusted content to limit its influence" (OWASP, 2025a), which only works for sources the prompt actually marks.
The fix is to stop treating the prompt as a separate piece of writing. Derive it from the blueprint field by field, tag each obligation so a checker can find it, and run that checker in CI next to the blueprint checker from Stage 3.
Blueprint field | Prompt Spec section | What the spec must say | Test kind |
Mission, goals (SC1–SC5) | 1 Purpose and success criteria | The mission verbatim; which goal outranks the others | Normal |
Inputs, with trust | 3 Inputs | Every untrusted source named and marked | Injection, one per source |
Responsibilities, workflow | 4 Task and required steps | The steps, in order, ending in "hand over if section 8 applies" | Normal, borderline |
Prohibited, with enforcement point | 5 Boundaries | Each rule tagged P1–P7 with "enforced by" and its layer | Gateway-enforced rules: one case each |
Top failure mode | 6 Examples | An example that invites it and the correct refusal | Borderline, flagged invites_scope |
Outputs, schema | 7 Output contract | The schema name and its fields | Every case checks the schema |
Hand-over cases, owners | 8 Uncertainty and escalation | Each case tagged H1–H6 with owner and packet | Must-escalate, one per case |
Evaluation pass bars | 9 Verification | The blueprint's release bar, not a new one | All |
The test: for each row, find the blueprint field and the spec section in your own agent and check they say the same thing. The rows where you have to think about it are the ones that drifted.
The trace, before and after
The checker for this stage, spec_trace.py, runs the Prompt Spec linter first and then asks the blueprint's questions. Run on the first draft, the linter passes and the trace doesn't:
$ python ../prompt-spec/prompt_spec_lint.py requirements-agent.prompt.v1.md
requirements-agent v1.0.0: passes (9 sections, tests linked)
$ python spec_trace.py requirements-agent.blueprint.json requirements-agent.prompt.v1.md
requirements-agent.prompt.v1.md (traced to requirements-agent.blueprint.json, spec v1.0.0, 4 test cases)
FAIL inputs 'backlog' text is untrusted in the blueprint but not marked untrusted in section 3
FAIL inputs 'requirement' text is untrusted in the blueprint but not marked untrusted in section 3
FAIL prohibited P1 'merge, edit or close a backlog record': no tagged rule in section 5
FAIL handover H3 'a search or read fails, times out or returns something unreadable' has no rule in section 8
FAIL tests H3 has no must_escalate case
FAIL tests no injection case arriving through 'requirement' text
FAIL tests no case invites extra scope (the blueprint's top failure mode)
FAIL output section 7 does not name the blueprint schema 'triage-note.v1'
verdict: FAIL (27 fail)
(Excerpt of stage4-output.txt: 8 of the 27 FAIL lines. The others are P2–P7, the remaining five hand-over rules and their test cases, the backlog injection case and the two gateway-enforced rules' cases.)
Some of the 27 are about form rather than substance, and it's worth being honest about which. The first draft did say, in one sentence, that the agent must not merge records, edit requirements, change priorities, contact the requester or accept requests. The trace fails those anyway, for two reasons that matter. They have no tag, so no test can be tied to them; and they don't say where they are enforced, so a reader of the prompt would reasonably believe the prompt is what stops a merge. It isn't: Stage 3 removed the merge scope from the agent's identity. A prompt that lists a rule without its enforcement point invites someone to delete the permission later because "the prompt already says it".
The substantive misses are the ones that cost something. Of the six hand-over cases, only the two regulatory ones were touched, by one sentence that named "compliance" and no packet; the other four were absent, including H3, the one the sandbox hit. Two of the three untrusted sources were unmarked. And the test pack had no case for the blueprint's top failure mode, a question that adds scope the requester didn't ask for, which is the failure Stage 1's interview surfaced and Stage 3's shadow week caught.
The second version was derived from the blueprint, row by row. Its boundaries section now reads, in part:
## 5 Boundaries
Each rule below is enforced outside this prompt; the prompt states it so your drafts don't attempt it, not so that it holds.
[P1] Never merge, edit or close a backlog record. Enforced by permission: the agent identity has no backlog.merge or backlog.write scope.
[P6] Never quote a request or requirement inaccurately. Enforced by gateway: the grounding check compares every quoted span with its source before a draft is saved.
and its escalation section carries the rule the sandbox needed:
[H3] If a search or read fails, times out or returns something unreadable: senior BA on duty, with what was searched and the error. Never report "no related requirements" after a failed search.
(Excerpts of requirements-agent.prompt.md; P2–P5, P7 and H1, H2, H4–H6 follow the same pattern.)
Traced again:
$ python spec_trace.py requirements-agent.blueprint.json requirements-agent.prompt.md
requirements-agent.prompt.md (traced to requirements-agent.blueprint.json, spec v2.0.0, 15 test cases)
verdict: PASS (0 fail)
The checker's own rules are tested the way Stage 3's were: break the passing spec in one place, a hand-over rule untagged, an enforcement point removed, requirement text no longer marked untrusted, the requirement-text injection case deleted, and confirm the trace fails. With seven new tests, the series' suite now has 28, and all pass.
The test: run your prompt linter, if you have one, and then answer by hand the question it can't: for each hand-over case in your design, which line of the prompt implements it and which test case proves it?
What the prompt is for, and what it isn't
Section 5 now opens with a sentence that matters more than it looks: each rule is enforced outside the prompt, and the prompt states it "so your drafts don't attempt it, not so that it holds". That is the right division of labour, and it changes how the prompt is reviewed.
A prohibition in a prompt is followed most of the time. A missing permission is followed every time. OWASP's guidance on excessive agency puts authorisation in the downstream systems rather than in the model (OWASP, 2025b), which is what Stage 3 did with the agent's scopes. So the prompt's job for P1–P5 is efficiency and clarity: a model that knows it can't merge records won't waste a step trying, and won't draft a note that says "merged as a duplicate". For P6 and P7, the gateway-enforced rules, the prompt's job is to make the grounding check's rejections rare, because two rejections on the same request trigger H6 and cost an analyst's time.
The hand-over rules are different. Most of them can only live in the prompt, because they ask the model to recognise a situation: a regulatory signal the screen missed, text addressed to the agent, a request spanning product lines. These are the rules where the prompt is the control, and therefore the rules that need the most test cases.
The test: mark each rule in your agent's instructions as stated (enforced elsewhere; the prompt only reduces attempts) or relied on (the prompt is the control). Every relied on rule needs at least one test case that runs on every change.
The test pack: one case for every obligation
The first draft's test pack had four cases: a normal request, a borderline one, one escalation and one injection. The derived pack has fifteen, and each exists because a blueprint field asked for it.
Kind | Cases | Why |
Normal | 2 | A clear duplicate; a request with nothing related |
Borderline | 4 | A real conflict; a request that invites extra scope; one draft that misquotes a requirement (P6); one question that quotes nothing (P7) |
Must-escalate | 6 | One per hand-over case, H1 to H6, including a forced search timeout for H3 |
Injection | 3 | One through each untrusted source: the request, a requirement's text, a backlog item |
Two of these are worth a closer look. The H3 case doesn't come from the request at all: it sets the search tool to time out, because the failure is in the environment, not the input. And the P6 and P7 cases test the gateway rather than the model: they feed a deliberately bad draft and expect the grounding check to refuse to save it, which proves the control the prompt claims exists.
These fifteen are the start of the test pack, not the whole of it. The blueprint's evaluation set, 200 closed requests with seeded slices, is Stage 12's job; the boundary and adversarial layers are Stage 11's. What Stage 4 guarantees is that nothing the blueprint promised is missing from the start. Prompt Regression Testing in CI covers running the pack five times on every change and diffing case by case.
How to check your own agent's instructions
Five checks, each answerable from the prompt, the design document and the test pack.
Every hand-over case in the design is a tagged rule in the prompt, with the owner and what to send.
Every input the design marks untrusted is marked untrusted in the prompt, including internal sources written by many people.
Every prohibition names its enforcement point, and none names "the prompt" for an action that a permission could stop.
The test pack has one must-escalate case per hand-over rule and one injection case per untrusted source.
There is a case that invites the agent's top failure mode, taken from the design's failure-mode list, and it runs on every change.
Tradeoffs and exceptions
Tagging every rule makes the prompt longer and a little less natural to read. The cost is real: a longer system prompt takes more tokens on every call. The alternative, a short prompt that can't be traced, has already cost one near-miss in the sandbox. Where the prompt grows past what is comfortable, move detail into tool descriptions or the retrieved context, and keep the tags; Context Engineering vs Prompt Engineering covers which layer a fact belongs in.
For a low-risk, read-only agent with no hand-over cases and no prohibitions beyond its permissions, the trace adds little; the Prompt Spec linter alone is enough. The trace earns its keep when the design has obligations that only the prompt can carry, which in practice means any agent with hand-over rules.
The trace also can't tell whether a rule is well written, only that it exists and is tested. A hand-over rule that is present but ambiguous will pass the trace and fail in production. That is what the test pack and, later, the evaluation set are for.
How Stage 4 goes wrong
The prompt is edited directly in production. A quick fix to the deployed prompt bypasses the spec file and the trace. Make the spec file the only source the deployment reads.
The blueprint changes and the spec doesn't. A new hand-over case is agreed in a review and added to the blueprint; the trace fails, and someone tags a placeholder to make it pass. Review trace fixes like code.
Tags without tests. The checker requires a must-escalate case per hand-over, but a case whose expected result is vague passes the checker and proves nothing. Every case needs an observable expectation.
The prompt becomes the control again. Months later, a permission is widened "because the prompt already forbids it". The enforcement points in section 5 are there to make that argument visibly wrong.
What to do next
Write your agent's prompt as a Prompt Spec if it isn't one already, then derive it again from the design document, row by row using the table above. Add tags for prohibitions and hand-over cases, and write the missing test cases: one per hand-over rule and one per untrusted source. Stage 5 gives the agent its tools, behind contracts tested against the blueprint's permissions; the prohibitions this stage tagged as "enforced by permission" are the ones Stage 5 has to prove.
Final takeaway
A prompt written from memory describes the agent the team remembers; a prompt derived from the blueprint describes the agent they designed. Derive it, tag every obligation, say where each rule is really enforced, and let a checker prove that every rule has a test before the prompt ships.
Sources
OWASP GenAI Security Project, LLM01:2025 Prompt Injection (2025a). Indirect prompt injection; separate and clearly denote untrusted content.
OWASP GenAI Security Project, LLM06:2025 Excessive Agency (2025b). Authorisation in downstream systems rather than the LLM.
Series navigation. Previous: Stage 3: Define Responsibilities, Permissions and Hand-overs. Next: Stage 5: Give It Tools, and Prove What Each One Does. The full stage list is on the Build an AI Agent From Scratch hub.
Frequently Asked Questions
How do you write instructions for an AI agent?
Derive them from the agent's design rather than writing them from memory. Every untrusted input, prohibited action and hand-over case in the design should appear in the instructions, each prohibition should say where it is actually enforced, and every rule should have a test case before the prompt ships.
Should an agent's system prompt list what it must never do?
Yes, but not as the control. A rule in the prompt stops the model from attempting an action most of the time; a missing permission or a check in code stops it every time. The prompt should state each prohibition and name the layer that enforces it.
How do you test an AI agent's prompt?
With a test pack that has a case for every obligation the design created: normal and borderline requests, one must-escalate case per hand-over rule, an injection case for every untrusted input source, and cases that invite the agent's most likely failure, each run several times.
Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.
