The AI Agent Observability Checklist: What to Record for Every Run

S. West

2 hours ago

14 min read

An agent you can't see is an agent nobody can evaluate, approve, debug or audit. Observability for agents means recording the fields that answer the questions people will ask after something goes wrong, before they ask them.

The example in this article is a composite scenario, built from patterns common to finance operations. It is not a single real organisation, and its figures are illustrative. The code is real: the checker output below was produced by running it over a simulated week of traces.

At the weekly cost review one Friday in September, the operations lead at a mid-sized distributor noticed that the duplicate-invoice investigator had become expensive. On Wednesday its cost per run had jumped from about 0.06 to 0.16 (illustrative prices), a factor of 2.7, and it had stayed there. The dashboard showed nothing else wrong: 40 runs that week, 40 completed.

The investigator was the agent from Designing Tool Contracts and Least Privilege for AI Agents: promoted to Level 2, drafting reversing entries that a clerk approves, behind a tool gateway. Its logging followed the pattern in our hands-on tutorial, Production Observability for Agents: a trace id per run, one event per tool step with latency and result size, the task text and the cost.

The finance director asked three questions. What changed? Which runs were affected? Did any of them put an entry in front of a clerk that was approved?

There was an obvious suspect. Prompt version 3.2.0 had shipped on Wednesday at 09:00, a canary on one legal entity, after passing the regression gate described in Prompt Regression Testing in CI. The team spent the next two days arguing about whether to roll it back. Nobody could settle it, because no run record said which prompt version it had used. The canary's scope lived in a deployment file, not in the trace. Nor could anyone answer the third question: the approval lived in the ERP, the run lived in the log, and nothing joined them.

Why "we log every step" feels like enough

The team's logs weren't careless. They answered the question engineers ask first: given this run, what happened in it? Pick a trace id and you could follow every tool call.

Incident questions are different. They are comparative and cross-system. Did runs on version A behave differently from runs on version B? Is this call the same as the last one? Which ledger entries came from these runs? Those can't be answered by reading one run more carefully. They need identifiers that let you group, compare and join: version fields, argument hashes, a flag for incomplete results and a reference to the record each run created. A log can be complete for debugging and still useless for diagnosis.

Google's SRE book frames monitoring around two questions: "what's broken, and why?" (Beyer et al., 2016). The team's cost alert answered the first. Nothing they recorded could answer the second.

The test: write down the last three questions anyone asked about your agent after something odd happened. For each, name the field you would group or join on to answer it. If the answer is "we'd read the logs", the field doesn't exist.

Four people ask four different questions

An agent's trace has at least four readers, and each needs something different from it:

Who asks	The question	The fields that answer it
The owner	Did something we changed cause this?	Prompt version, agent version, model id, configuration hash, change events
QA	Is it looping, and is quality holding?	Arguments hash per tool call, retries, result completeness, online evaluation scores
Security	Did it try something it shouldn't, on whose data?	Tool status including refusals, trigger, tenant, credential identity
The auditor	Who approved what this agent produced?	Approval decision, approver, time to decide, reference to the record created

Most logging is designed by the engineer who built the agent, so it serves the engineer's question and often the owner's. The auditor's question is usually the one nobody planned for, because it spans two systems. In the composite, the drafted journal entry had an id in the ERP and the run had an id in the log, and neither recorded the other.

This is a discovery gap, not a tooling gap. The questions were predictable on the day the agent was scoped. A discovery move would have caught it: at intake, ask each of these four people what will you need to know about a run six weeks after it happened? and write each answer down as a field.

The test: take one real run from last month and try to answer all four rows from the trace alone. Count the rows you can't.

The incident, rebuilt from a trace that could answer

To show the difference, we simulated the composite week twice from the same runs: once as the team's original logging recorded it, and once recorded against the checklist below. The checker reads the checklist as data (each field, the question it answers and who asks it) and reports what a trace file can't answer. Here's how a field is declared:

RUN_FIELDS = [
    ("run_id",         "Which run is this complaint or draft about?",                 "owner"),
    ("prompt_version", "Did the prompt change cause this?",                           "owner"),
    ("config_hash",    "What tools, limits and autonomy level did it have then?",     "auditor"),
    # ... 14 run fields, then model step, tool step, human and quality fields
]
LOOP_REPEAT_LIMIT = 3          # same tool + same arguments this many times in one run

On the original logs (excerpt):

  run        prompt_version        missing  40/40  owner    Did the prompt change cause this?
  model step no records of this kind at all
  tool step  args_sha              missing 190/190 QA       Was it the same call again (a loop)?
  tool step  result_meta           missing 190/190 QA       Was the result complete (row count, truncated flag)?
  human      record_ref            missing  21/21  auditor  Which record in the system of record came from this run?

PII CHECK
   40 unredacted email value(s) in run.task

LOOP CHECK (same tool + same arguments >= 3 times in one run)
  cannot run: tool steps carry no argument hash, so a repeat looks like a new call

On the same week, recorded against the checklist (excerpt):

LOOP CHECK (same tool + same arguments >= 3 times in one run)
  r-0916-03  search_invoices x9  prompt 3.1.0  E2  outcome=abandoned truncated results: all
  r-0916-06  search_invoices x4  prompt 3.1.0  E2  outcome=completed truncated results: all
  r-0916-08  search_invoices x9  prompt 3.2.0  E1  outcome=abandoned truncated results: all
  r-0917-03  search_invoices x9  prompt 3.1.0  E3  outcome=abandoned truncated results: all
  r-0917-06  search_invoices x4  prompt 3.1.0  E3  outcome=completed truncated results: all
  r-0918-03  search_invoices x9  prompt 3.2.0  E1  outcome=abandoned truncated results: all
  r-0918-06  search_invoices x4  prompt 3.2.0  E1  outcome=completed truncated results: all
  r-0918-08  search_invoices x9  prompt 3.2.0  E3  outcome=abandoned truncated results: all
  looping runs by prompt version, since 2026-09-16: 3.1.0: 4/10, 3.2.0: 4/14

COST PER COMPLETED TASK (all run cost / completed runs)
  2026-09-14  runs 8  completed 7  cost/run 0.060  cost/completed 0.069
  2026-09-16  runs 8  completed 5  cost/run 0.162  cost/completed 0.260  ALERT

PROHIBITED-ACTION ATTEMPTS REACHING ENFORCEMENT
  r-0917-04 step 7: post_journal -> not_permitted

HUMAN LAYER FOR FLAGGED RUNS
  8 flagged runs; 3 produced a record a person decided on
  r-0916-06 -> JE-D422  edited by clerk-3 in 103s
  r-0917-06 -> JE-D430  edited by clerk-3 in 105s
  r-0918-06 -> JE-D438  edited by clerk-3 in 159s

Read the loop block first. Since Wednesday, runs on the old prompt looped at 4 of 10, and runs on the new one at 4 of 14. The counts are small, but the conclusion doesn't rest on the ratio: the old prompt loops too, so rolling back would have changed nothing. Every looping run repeated an identical search_invoices call, and every one of those results was flagged as truncated. Retries on each call were zero, so it was the agent repeating itself, not the gateway retrying.

The cause wasn't in the agent's change log at all. At 07:30 that Wednesday, an ERP patch had cut the search API's default page size from 100 to 20 (in the composite). For suppliers with more than 20 open invoices, the agent got a partial list, didn't find the pair, and searched again with the same arguments, nine times, until the run gave up. The give-up path returned a polite "no duplicate found", which the original logs recorded as a completed run. That's why the dashboard showed 40 of 40: five runs since Wednesday had given up and three had found a way round, all reported as success. Cost per run rose 2.7 times; cost per completed task, measured honestly, rose 3.8 times.

The finance director's third question now has an answer too. Three looping runs fell back to a search by amount and produced drafts. All three were edited by the clerk before approval. And the checker found something nobody had asked about: on Thursday, one run tried to post a journal entry and the gateway refused it. The original log showed a post_journal step, but not whether it had been refused or had gone through.

The discovery move. The missing field that mattered most, result completeness, is the one an intake question would have produced: for each tool, what does it return when the result is incomplete, and how would the trace show it? External systems change without telling your change board. Recording what came back is the only way their changes show up in your data.

The Agent Observability Checklist

This is the checklist the code enforces. It's built to be copied: each field states what to record and the question it settles. The "who asks" column tells you whom to involve when you decide whether a field is optional for your agent. The diagram shows how the layers fit.

Per run

Field	Record	Question it settles	Who asks
Run id	Unique id, propagated to every step and every record the run creates	Which run is this about?	Owner
Agent id and version	The agent's own identity, not a shared account, and its release	Which agent acted, in which release?	Security
Prompt version	Every prompt and template version used in the run	Did the prompt change cause this?	Owner
Model	Requested and returned model id	Which model produced it?	QA
Configuration hash	Hash of tools, limits, step cap and autonomy level	What was it allowed to do at the time?	Auditor
Trigger, user and tenant	What started the run, for whom, on whose data	Who or what started it?	Security
Start and end	Timestamps; derive latency	When, and how long?	Owner
Outcome	Completed, escalated, failed or abandoned, with a reason code; never "completed" for a run that gave up	Did it finish the job?	Owner
Tokens and cost	Tokens in and out; cost	What did this run cost?	Owner

Per step

Field	Record	Question it settles	Who asks
Model call	Context manifest hash, finish reason, tokens, a reference to the stored output	What was in the context window?	QA
Tool call	Tool, arguments hash (plus redacted arguments), status including refusals, error code, duration, retry count, idempotency key	Same call again? Refused? Retried?	QA, security
Tool result	Row count, truncated or has-more flag, size	Was the result complete?	QA
Decision point	The action chosen and a reason code	Why did it escalate or stop?	Owner
Retrieval	Document ids, scores, source freshness	What evidence did it use?	QA
Memory	Ids of records read and written	What did it remember, and from where?	Security

Human layer

Field	Record	Question it settles	Who asks
Approval	Requested, then approved, edited or rejected; approver; time to decide; link to the edit	Who approved this, and did they change it?	Auditor
Record reference	The id of every record the run created in the system of record	Which entries came from this run?	Auditor
Escalation	Reason, who picked it up, how it was resolved	What happened to the cases it handed over?	Owner

Quality layer

Field	Record	Question it settles	Who asks
Online evaluation	Score, rubric or judge name and version	Is quality holding?	QA
Sampled review	Whether the run was sampled; the reviewer's verdict	Does a person agree with the score?	QA
User feedback	Explicit feedback linked to the run id	Did the user think it worked?	Owner

Change events (one record per change, outside the runs): what changed (prompt, model, tools, permissions, autonomy level, step cap), from and to, scope, who, when and the approval reference.

A context manifest is the list of what was placed in the model's context window for a call (each item's id, source and hash). Storing its hash per call lets you prove two runs saw the same context without storing the context itself. Context Engineering: A Practitioner's Guide shows how to build one.

The test: run the checklist against one day of your traces with the checker in the code folder, or by hand. Any field missing from every record is a question you have already decided not to be able to answer.

Alerts: page on what someone must act on

The SRE book's rule for paging applies unchanged: "Every page should be actionable" (Beyer et al., 2016). Six alerts earn that for agents. Thresholds below are illustrative starting points; set yours from a baseline week.

Rate change. Error, fallback, escalation and abandon rates compared with a trailing baseline. The checker uses 5 percentage points. A falling escalation rate deserves the same attention as a rising one: as Evaluating AI Agents notes, a reviewer edit rate drifting towards zero can mean the agent improved or that reviewers stopped looking. Watch time to decide alongside it.
Cost per completed task, not cost per run. Cost per run hides runs that consume budget and finish nothing. OWASP lists "Denial of Wallet" under unbounded consumption and recommends that you "continuously monitor resource usage and implement logging to detect and respond to unusual patterns of resource consumption" (OWASP, 2025). The checker alerts at 1.5 times baseline.
Loop detection. The same tool with the same arguments hash three times in one run, or any run reaching the step cap. This needs the arguments hash; without it, a repeat looks like a new call.
Prohibited-action attempts that reach enforcement. The target is zero attempts, and every attempt alerts. A refusal at the gateway means the control worked, and it also means something (a prompt change, injected text, a new kind of input) pushed the agent towards a forbidden action. The audit record from the tool-contracts gateway is the source.
Configuration changes. Any change to prompts, tools, models or autonomy notifies the owner and is recorded as a change event, so every later comparison can split on it.
Trace completeness. Alert when a required field goes missing. Telemetry breaks quietly in a refactor, and you'll find out during the next incident unless something checks.

The test: for each alert you run today, write down who gets it and the first thing they'd do. An alert with no first action is noise; drop it or give it an owner.

Retention and privacy: record less, keep it shorter

Agent traces collect personal data almost by accident: the original logs above carried a supplier contact's email in every one of 40 task descriptions. The OpenTelemetry conventions treat message content as opt-in for this reason. Content attributes are marked Opt-In, and for memory content the spec says instrumentations "SHOULD NOT capture this attribute by default" (OpenTelemetry, 2026).

Where traces hold personal data under the GDPR, two principles set the defaults: data must be "adequate, relevant and limited to what is necessary" for the purpose (data minimisation), and kept in identifiable form "for no longer than is necessary" (storage limitation) (Regulation (EU) 2016/679, Art. 5(1)(c) and (e)). This article isn't legal advice; your data protection officer sets the actual periods. In practice:

Record hashes and references by default. Arguments hash, context manifest hash and output reference answer most questions without the content.
Redact before export, and test the redaction. The checker's PII patterns catch the obvious cases (email, IBAN, phone). They aren't a data-loss-prevention product.
Keep raw content in a separate, access-logged store with a shorter retention period than the metadata.
Agree a retention period per record class with the data owner: run metadata, step content, approval records. Approval records often carry an audit obligation of their own and may need to outlive the rest.
Test deletion. A retention policy nobody has seen run is a hope.

The test: search one day of your trace store for an email pattern. If anything comes back, redaction happens later than export, or not at all.

Using the OpenTelemetry GenAI conventions

If you emit traces through OpenTelemetry, use its generative AI semantic conventions for the fields they cover. As of 9 October 2026 they are marked Status: Development, and they have moved out of the main semantic conventions site into their own repository. The old opentelemetry.io page now reads "GenAI semantic conventions have moved to the OpenTelemetry GenAI semantic conventions repository" (OpenTelemetry, n.d.-a). OpenTelemetry's maturity definitions say a Development component "SHOULD NOT be used in production" and "MAY be removed without prior notice" (OpenTelemetry, n.d.-b). That's a reason to adopt them carefully, not to avoid them.

They cover a useful core: invoke_agent and execute_tool spans, gen_ai.agent.id and gen_ai.agent.version, gen_ai.prompt.name and gen_ai.prompt.version, gen_ai.request.model, gen_ai.tool.name and gen_ai.tool.call.id, token usage, and evaluation attributes. They don't cover several fields this checklist needs: a configuration hash, a context manifest hash, an arguments hash, retry counts, result completeness, approval decisions or a link to the record created. Put those in your own attribute namespace.

Pin the version you implement, map conventions to your field names in one place, and add a test that fails when a required attribute disappears. Then a rename upstream costs one mapping change, not a hunt through every service.

The test: list the checklist fields your instrumentation emits today and mark which use a gen_ai.* name and which use your own. Anything with neither isn't being recorded.

Tradeoffs and exceptions

Choice	Gains	Costs	Use when
Hashes and references instead of content	Privacy; smaller traces	Some debugging needs the content store	Default for every agent
Full content capture	Fast debugging and replay	Personal data in traces; storage cost	Pre-production, or a short-retention restricted store
Every run traced in full	Nothing missed	Volume and cost	Agents that write or touch money, customers or personal data
Sampled step detail, full run records	Cheaper	A rare loop can go unseen	High-volume, read-only agents
Strict alert thresholds	Early detection	Pages that teach people to ignore pages	After a baseline week, then tighten

Exceptions. A read-only internal agent over non-sensitive data can sample step detail, but should keep full run records, because cost per completed task and outcome rates need every run. A prototype needs less, until the day it gets its first write tool; add the human layer then, not after the first audit request.

How agent observability goes wrong

Give-up exits logged as success. The most expensive line in the composite. An agent that gives up politely must still be recorded as abandoned.
Version on the deployment, not the run. Canaries and gradual rollouts make deployment state useless for diagnosis. Put the version on every run.
No join key to the system of record. Then nobody can answer the auditor, and every drafted record is orphaned.
Only successes logged. Refused calls are the early warning.
External changes treated as invisible. Your dependencies' changes reach you only through what your tools record about their results.

Who owns what

The agent's owner signs off the field list and the alert routing, and receives configuration-change notifications. The platform team owns instrumentation and the completeness check. QA owns the loop, rate and quality alerts and runs the checker in CI against a recorded trace sample, so a missing field fails a build rather than an incident. Security receives every prohibited-action alert. The data owner and data protection officer set retention per record class. A promotion up the autonomy ladder should require the checklist live and its alerts reaching a named person.

What to do next

Start with the hands-on tutorial if you have no tracing yet. If you do, export one week of traces, run them against the checklist (the checker works on any JSON-lines file with run, step and human records), and fix the fields in this order: outcome reasons, prompt and configuration versions, arguments hash and result completeness, then the record reference. Add the trace-completeness alert last, so the gaps you just closed stay closed.

Final takeaway

Logging that lets you read one run isn't observability. The test is whether the trace can answer the owner's, QA's, security's and the auditor's questions about runs you haven't looked at yet. Decide those questions at intake, give each one a field, and alert when a field goes missing.

Sources

OpenTelemetry, Semantic Conventions for Generative AI, GitHub repository, accessed 9 October 2026. Documents marked "Status: Development"; agent spans (invoke_agent, execute_tool); attributes including gen_ai.agent.version, gen_ai.prompt.version, gen_ai.tool.call.id; content attributes Opt-In, memory content "SHOULD NOT" be captured by default.
OpenTelemetry (n.d.-a), Moved: Generative AI semantic conventions, opentelemetry.io, accessed 9 October 2026. Redirect notice pointing to the new repository.
OpenTelemetry (n.d.-b), Maturity levels, OpenTelemetry Specification, accessed 9 October 2026. Definition of Development maturity.
OWASP GenAI Security Project, LLM10:2025 Unbounded Consumption, 2025. Denial of Wallet; monitoring resource usage; rate limits, timeouts and throttling.
Beyer, B., Jones, C., Petoff, J. and Murphy, N. R. (eds.), Site Reliability Engineering, Chapter 6: Monitoring Distributed Systems, O'Reilly, 2016. "What's broken, and why?"; "Every page should be actionable."
European Union, Regulation (EU) 2016/679 (General Data Protection Regulation), Article 5(1)(c) data minimisation and 5(1)(e) storage limitation.

Frequently Asked Questions

What should you log for every AI agent run?

At minimum: a run id, the agent and its version, the prompt version, the model id, a hash of the configuration (tools, limits, autonomy level), the trigger and tenant, start and end times, the outcome (completed, escalated, failed or abandoned), tokens and cost. Per step, record a context manifest hash for model calls and, for tool calls, the tool, an arguments hash, status, retries and whether the result was complete.

Which alerts matter most for AI agents in production?

Changes in error, fallback and escalation rates against a baseline; cost per completed task rather than cost per run; loops, such as the same tool called with the same arguments three times in one run; every prohibited action that reaches enforcement; every configuration change; and missing trace fields.

Are the OpenTelemetry GenAI semantic conventions stable?

No. As of 9 October 2026 they are marked Status: Development and have moved to their own repository. Use their attribute names where they exist, pin the version you implement, and keep your own fields, such as configuration hash and approval links, in your own namespace.

Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.