Build an AI Agent, Stage 13: Observability That Answers Four Readers From One Trace
A run record is finished when the people who will question it can get their answers from it. Write each reader's question as a query before release, name the system that writes every field the query needs, and fail the build if any query comes back unanswered.
The example in this series is a composite scenario, built from patterns common to requirements intake in regulated organisations. It is not a single real organisation, and its figures are illustrative: the records in this stage are composite, not measured. The code is real: every output block below was produced by running it.
Version 2.3 of the Requirements Agent's question skill was rolled back at 11:00 on a Monday, after Stage 12 showed that its friendlier questions were not better and that some of them were leading. By the end of that afternoon, four people had asked four questions.
The BA practice lead, who owns the agent, asked which requests 2.3 had triaged. The QA lead asked whether analysts really had been changing more of its questions. A security engineer asked whether anything in those two weeks had acted outside the agent's permissions. And the compliance intake lead asked the question that mattered most: which of 2.3's questions had been sent to brokers, and by whom, because a leading question that reached a broker could have put scope into a request the broker never meant to ask for.
The run log could answer none of them. It was the workflow trace from Stage 9, one line per step saying whether it succeeded, with the run's id, request, times, outcome and cost added when the agent went into the pilot. Two people spent three days answering by hand, matching the text of sent emails against the drafts in the review queue and reading versions off the deployment history. Their answer to the first question was that 2.3 ran from the promotion until the rollback. That answer was wrong.
This is Stage 13 of Build an AI Agent From Scratch. Stage 12 built the evaluation that decides whether a new version is better. This stage builds the run record, and the test that proves it answers the people who will read it.
Why a complete log still can't answer
The pilot's log was complete in the sense its builder meant: pick a run, and every step is there. That answers the engineer's question, what happened in this run? None of the four questions was about one run. The owner's groups runs by version, and the security engineer's by identity. The QA lead's joins reviewers' decisions to runs. The compliance lead's joins three systems: the emails sent, the drafts in the review queue and the runs that wrote them. A log can be complete for debugging and useless for all four, because the fields you group and join on were never written.
The AI Agent Observability Checklist lists those fields and the four readers who need them. This stage applies it to the Requirements Agent, and finds the part a checklist alone does not show: of the four fields that would have answered the four questions, the agent's own code writes one. The tool gateway and the review queue write two more, and the agent's team could add those because every draft passes through them. The mail system writes the fourth, and it belongs to another team.
The test: for each field your agent's readers would group or join on, write down which system writes it and who owns that system. Any field written by a system your team doesn't own is a requirement on another team, and it needs to be raised as one.
Four readers, four queries, four writers
The discovery move is to ask each reader for their question before release, in their words, and turn it into a query against a sample of recorded runs. A question in a design document can be agreed and forgotten. A query either answers or names the field it is missing.
The four questions, the fields their queries need and the system that writes each one, all recorded as data in requirements-agent.observability.json:
Reader | The question | Fields the query needs | Written by |
Owner (BA practice lead) | Which runs used question skill 2.3, including any after the rollback? | The skill version each run loaded; the worker that ran it | The agent |
QA lead | Did analysts change more of 2.3's questions than 2.2's? | The run id on each review decision | The review queue |
Security engineer | Did any run act under another identity, or try a call it was refused? | The credential and environment of each run; a step status that tells refused from failed | The tool gateway |
Compliance intake lead | Which questions sent to requesters came from 2.3 drafts, and who sent them? | The draft ids carried by each sent email | The mail system |
The owner's field has a trap in it. The deployment history says which version was deployed, and when. The run record has to say which version the run loaded, which differs whenever a worker caches a skill bundle, a run spans a deploy or a canary splits traffic. Only the run knows what it ran.
The compliance lead's field took the longest to get. The query needed each sent email to carry the ids of the drafts in it, and the analysts send questions from a screen the mail system's team owns, so it took a change request to that team. Distributed tracing settled the same rule for HTTP services: the W3C Trace Context standard requires, at a minimum, that tracing tools "propagate the traceparent and tracestate headers and guarantee traces are not broken" (W3C, 2021). A trace breaks at the first system that drops the id.
The test: ask each of your agent's readers for the question they would ask six weeks after an incident, and write it as a query against last week's runs. Each query that can't run names a field, and the field names an owner.
The two weeks, recorded both ways
To show the difference, the composite's two weeks are recorded twice from the same runs: as the pilot's log recorded them, and with every field the four queries need. The checker for this stage, trace_check.py, reads the queries and the blueprint's six operational metrics from the spec and checks that every field they need is in every record before it runs anything. If a field is missing, it names the field, the number of records and the system that writes it.
On the pilot's log:
$ python trace_check.py v1
trace v1: 185 runs, 2026-09-14 to 2026-09-30; review queue 293 decisions; mail 165 messages
FOUR READERS
owner BA practice lead Which runs used question skill 2.3, including any after the rollback?
CANNOT ANSWER: runs.versions.question_skill missing in 185/185 records (written by: agent)
QA QA lead Did analysts change more of 2.3's questions than 2.2's?
CANNOT ANSWER: review_queue.run_id missing in 293/293 records (written by: review queue)
security security engineer Did any run act under an identity other than the agent's, or try a call it was refused?
CANNOT ANSWER: runs.credential missing in 185/185 records (written by: tool gateway)
auditor compliance intake lead Which questions sent to requesters came from 2.3 drafts, and who sent them?
CANNOT ANSWER: mail.draft_refs missing in 165/165 records (written by: mail system)
BLUEPRINT METRICS (6)
runs per day 14.2
hand-over rate by reason CANNOT COMPUTE: runs.handover_reason missing in 17/17 records (written by: agent)
grounding-check failures per day CANNOT COMPUTE: no grounding_check steps recorded (written by: tool gateway)
tool refusals per day CANNOT COMPUTE: steps.status records error/ok; needs ok/refused/failed to tell a refusal from an outage (written by: tool gateway)
cost per request 0.066
p95 run time 203 s
result: 0 of 4 readers answered, 3 of 6 metrics computable: FAIL
None of the four readers gets an answer, and three of the blueprint's six operational metrics, the ones Stage 3 wrote down, can't be computed at all. The log recorded every tool step's status as ok or error, so a call the gateway refused looks exactly like a search that timed out.
On the same two weeks, recorded against the spec:
$ python trace_check.py v2
trace v2: 185 runs, 2026-09-14 to 2026-09-30; review queue 293 decisions; mail 165 messages
FOUR READERS
owner BA practice lead Which runs used question skill 2.3, including any after the rollback?
question skill 2.2: 38 runs 2.3: 147 runs
2.3 after the rollback at 11:00 on 2026-09-28: 12 runs, worker w-2, last 2026-09-29 15:30
QA QA lead Did analysts change more of 2.3's questions than 2.2's?
question skill 2.2: analysts edited or rejected 8/56 questions (14%)
question skill 2.3: analysts edited or rejected 51/195 questions (26%)
security security engineer Did any run act under an identity other than the agent's, or try a call it was refused?
refused calls: 0 failed calls: 5 search_requirements
runs under another identity: 3 (eng-14 from laptop, 2026-09-25), 5 drafts, 3 approved by an analyst
auditor compliance intake lead Which questions sent to requesters came from 2.3 drafts, and who sent them?
messages to requesters carrying a 2.3 draft: 131, sent by 4 analysts
with a question the calibrated judge flags as leading, sent unedited: 19 (sent by ba-02, ba-05, ba-07, ba-11)
first: REQ-4118 (M-9016), REQ-4123 (M-9021), REQ-4130 (M-9027)
BLUEPRINT METRICS (6)
runs per day 14.0
hand-over rate by reason 9% (search failed 5, regulatory signal 4, more than one product line 4, text addressed to the agent 4)
grounding-check failures per day 0.77
tool refusals per day 0.00
cost per request 0.064
p95 run time 203 s
result: 4 of 4 readers answered, 6 of 6 metrics computable: PASS
(Both from stage13-output.txt. The records are composite and illustrative; every count is computed by the script.)
Read the owner's answer first, because it corrects the hand-built one. Version 2.3 did not stop at the rollback. One of the two workers had cached the skill bundle and kept serving 2.3 for twelve more runs over two days, until it was restarted on Wednesday morning. The deployment history said 11:00 on Monday because that is when the rollback was deployed. Only the runs could say when it took effect.
The other three answers each changed what happened next. The QA lead was right: analysts edited or rejected 26% of 2.3's questions, against 14% for 2.2. The security engineer's answer found something nobody had asked about. On the Friday, an engineer investigating 2.3 had re-run three requests from a laptop under a personal account; those runs wrote five drafts into the live review queue, and analysts approved three of them. The pilot's log recorded them as three more agent runs, which is also why it counts 14.2 runs a day where the service ran 14.0. And the compliance intake lead got a list: 19 messages in which a question the calibrated judge flags as leading went to a broker unedited. Those 19 requests went back to their analysts to check whether the broker's reply had added scope.
The test: take the last incident or rollback on your agent and answer its first question from the trace alone, then compare with what the deployment history says. If the two can differ and your trace can't tell you which is right, the trace is recording what you deployed, not what ran.
A run record the build can fail
The four queries and the six metrics are the stage's test. The checker exits non-zero if any reader goes unanswered or any metric can't be computed, so it runs in CI against a recorded sample of runs from the sandbox. The spec says not only which fields must exist but, where it matters, which values they may take: a step status of error fails, because it can't tell a refusal from an outage, and the security reader needs to.
Seven new tests break a passing trace in one place each: one run with no version, one step recorded as error, one email with no draft ids, and each is caught and names its writer. With them the series suite has 95 tests, and all pass.
The test is only as good as its sample. A sample with no hand-overs can't show that the hand-over reason is recorded, so it needs a run of every kind the blueprint names. The Stage 11 suite already produces them: its boundary layer has a case for each of the six hand-overs, and its adversarial layer has injection requests that ask for actions the agent must never take.
The test: remove one field from one record in your trace sample and run your observability check. If the build still passes, nothing is enforcing the field, and it will disappear the first time someone refactors the logging.
Recording what is needed, and no more
Requests carry personal data, so a run record that answers four readers could easily become a copy of every request. This one holds ids, versions, statuses, hashes of tool arguments and references to the records the run created. It holds no request text, draft text or requester names: no query needs them, and the review queue and mail system already keep them under their own retention rules and access controls.
For high-risk AI systems, the EU AI Act requires that the system "technically allow for the automatic recording of events (logs) over the lifetime of the system" (Article 12(1)). Whether or not a given agent falls under that article, the auditor's query is the practical form of it: can a person reconstruct who approved what, from the records alone, long after the run? If you emit traces through OpenTelemetry, its generative AI conventions cover part of the record, including invoke_agent and execute_tool spans and gen_ai.agent.version (OpenTelemetry, accessed 2026). They are still marked Development and have nothing for a review decision or the draft ids on a sent email; those go in your own namespace.
The test: list every free-text field in your run record and name the reader whose query needs it. Any field no query needs is personal data you are keeping for no one.
Building the run record: a method
Ask each reader for their question. The owner, QA, security and the auditor, at least. Done when each question is written in the reader's own words.
Write each question as a query. Run it against a sample of recorded runs. Done when every query either answers or names the field it is missing.
Name the writer of every field. The agent, the tool gateway, the review queue, the mail system or any other. Done when every field has a writer and an owning team, and fields owned by other teams have a change request.
Add the blueprint's operational metrics. Done when every metric the blueprint lists can be computed from the same trace.
Make the queries a build gate. Done when the check runs in CI on a sample containing every kind of run, and removing any needed field from one record fails the build.
Tradeoffs and exceptions
Approach | Gains | Costs | Best suited for | Warning signs |
Log every step, answer by hand | Cheap to build | Days per question; answers can be wrong | Prototypes | "We'd have to read the logs" |
Checklist of fields | Fast start; covers common readers | Doesn't show who writes each field | A first production agent | Fields nobody queries |
Readers' queries as a build gate | Every question answered before it is asked | Writing the queries; other teams' changes | Agents whose output reaches people | Queries written once and never revisited |
Full content capture | Anything can be answered later | Personal data kept for no reader | Short-lived debugging, with consent | Request text in every span |
Queries drift with the agent. A new tool, a new hand-over reason or a new reviewer gate brings a new question, and a reader who joins later, such as a data protection officer, brings theirs. Review the queries at every release that changes what the agent can do, the same way the blueprint is reviewed. The run record doesn't have to answer everything itself, either. If the review queue already reports edit rates, the QA lead only needs the run id on each decision to split that report by version. The record has to hold the keys that let every system's answers be joined.
How Stage 13 goes wrong
The deploy log stands in for the run record. It records what was deployed, not what ran. Record the version each run loaded, and the worker that ran it.
Refusals are logged as errors. Security can't tell the gateway working from the gateway failing. The Stage 5 tools already tell a refusal from a timeout; record them as different statuses.
Only the agent's code is instrumented. The auditor's question needs a field the mail system writes. Find it by writing the query, and raise it with that team before release.
Someone else's runs look like the agent's. Runs from a laptop or a test harness, under a person's account, land in the same queue. Record the credential and environment of every run, and alert on any that isn't the agent's own.
What to do next
Ask your agent's four readers this week for the question each would ask after an incident, write each one as a query against last week's runs, and see which come back unanswered. For each missing field, name the system that writes it, and raise the change with the team that owns it. Stage 14 deploys the agent: a versioned release, a staged rollout, and a rollback rehearsed before the canary starts.
Final takeaway
A run record is finished when the people who will question it can get their answers from it. Ask the owner, QA, security and the auditor for their questions before release, write each as a query, and name the system that writes every field it needs, including the systems your team doesn't own. Then make the queries a test the build can fail, and the next rollback's questions are answered in minutes, from the runs, correctly.
Sources
World Wide Web Consortium, Trace Context, W3C Recommendation, 23 November 2021. Tracing tools "MUST propagate the traceparent and tracestate headers and guarantee traces are not broken."
European Union, Artificial Intelligence Act, Article 12: Record-keeping. Automatic recording of events over the lifetime of high-risk AI systems.
OpenTelemetry, Semantic Conventions for Generative AI, GitHub repository, accessed 14 October 2026. Agent spans (invoke_agent, execute_tool) and gen_ai.agent.version, marked Status: Development.
Series navigation. Previous: Stage 12: Calibrate the Judge Before You Read Its Score. Next: Stage 14, a versioned release with a staged rollout and a rehearsed rollback (coming). The full stage list is on the Build an AI Agent From Scratch hub.
Frequently Asked Questions
What should an AI agent's run record contain?
Whatever its readers need to answer their questions without reading logs by hand. For most agents that means the versions each run actually loaded, the identity it acted under, each tool call's status with refusals told apart from failures, a reference to every record it created, and the reviewer's decision on each one.
Who are the readers of an AI agent's trace?
At least four: the owner, who asks whether a change caused something; QA, who asks whether quality is holding; security, who asks whether it acted under the right identity and permissions; and the auditor, who asks who approved what it produced. Each needs different fields.
How do you test AI agent observability?
Write each reader's question as a query against a sample of recorded runs before release, and fail the build if any query cannot be answered. Do the same for every operational metric the agent's blueprint lists.
Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.
