Observability for AI Agents: Instrument the Trajectory, Not the Request

A support agent answers in 1.2 seconds, returns a well-formed JSON body, and the monitoring dashboard stays green. The customer's refund was applied to the wrong order. Nothing in your telemetry noticed, because nothing in your telemetry was looking at the right thing: the HTTP layer recorded a success, the duration was healthy, and the log line says completed. The failure was not an exception. It was a decision.

That is the whole problem with observability for agents. Classical observability is built on three assumptions that hold for a deterministic service and break for a system whose next step is chosen by a model. This article is about the instrumentation that replaces them: what one unit of observation is, what to capture, how to sample traces that can be expensive and long, what to alert on when the failure mode is a plausible wrong answer, and how to get from a dashboard to a diagnosis.

Scope, because this area is full of overlapping vocabulary. This is about telemetry — the spans, attributes, and rollups that let you reconstruct and debug what an agent did. It is not about measuring whether the output was good (that is the discipline covered in the companion piece on measuring the quality of an AI system), and it is not about the immutable audit trail with retention and tamper-evidence requirements (that lives here). The three share plumbing and must not share a schema, for reasons the payload section below gets into.

Examples are Python and OpenTelemetry Collector YAML. The instrumentation shape is the point, and it maps onto any SDK.

Why agents break the observability model

Determinism is gone, so "reproduce the bug" is the wrong verb. A service bug is reproducible: same request in, same code path, same crash. An agent given the same input twice may take a different path, call a different tool, and produce a different answer, because sampling, provider-side routing, and model updates all introduce variance you do not control. So the debugging workflow cannot be "reproduce and step through". It has to be "read the trajectory of the run that failed, then replay the frozen inputs many times and look at the distribution". That is a different instrument, and it changes what you must capture.

Failure is not an exception. A service either throws or it does not. An agent returns 200 OK with a confident, well-formed, wrong answer, and your error rate does not move. Every signal that classical observability is built on — status codes, exception counts, error rates — is blind to the failure that matters. The signal has to be manufactured: a verifier's verdict, an abstention, a schema check, a reconciliation against a second source. If you have not instrumented that, you have instrumented nothing that detects the interesting regression.

The unit is not the request. A user request can be one model call or forty, spread across tool invocations that touch three systems, with a session that outlives the HTTP connection. If your root span is the HTTP request, you get a trace that ends before the interesting part begins, and you cannot answer the only question that matters in an incident: which step first went wrong, and what did the model see when it decided.

And the payload is the debugging artifact. Classical telemetry deliberately omits payloads — you log a status code and a route, not the request body, because the body is noise for 99 percent of incidents. For an agent it is the opposite: the prompt, the retrieved context, and the tool arguments are the incident. You cannot debug an agent trajectory without them, and you cannot store them naively without creating a compliance problem. That tension is the central engineering constraint of agent observability, and it is why "just log everything" is not a plan.

The unit of observation: the task trajectory

The root span should be the task — the unit of work the user asked for — not the transport request. Under it, one span per decision and one span per effect:

• task (invoke_agent) — the whole trajectory. Carries the task identity, the goal, and the aggregate outcome.
• turn / model call (chat) — one model invocation, with the request parameters, provider, model version, and token usage.
• tool execution (execute_tool) — one tool call, with the tool name, the arguments, and whether the effect succeeded. This is where the agent touched the world.
• retrieval — one search: the query, the ids returned, and their scores. Not just "retrieval happened".
• verifier / guardrail — one check, with its verdict and its confidence. This is the span that turns a silent failure into an observable one.

That nested model has a limit, and it is worth knowing before an agent grows into it. Nesting assumes the trajectory is short-lived enough to be one trace, which is true for a request-response task and stops being true in two cases. A long-running agent that loops for hours produces a single enormous trace: collectors hold it in memory while waiting to decide, the tail-sampling buffer drops the oldest traces when it overflows, and there is a configured ceiling beyond which a trace is discarded outright rather than sampled. Decide that ceiling carefully, because for a three-hour agent the trace that matters most is the one guaranteed to exceed it. A human-in-the-loop pause has the same shape from the other direction: the process stops, the trace ends, and when a person answers six hours later the work resumes with no parent to attach to.

Both cases take the same remedy, and it is a part of the OpenTelemetry specification rather than a trick: span links, which connect spans that are related without being parent and child. A long loop becomes a sequence of linked segments — a trace per turn, or per hour, or per resume — with each segment pointing at the one before it, so a reader can still walk the whole trajectory without any single trace having to be enormous. The practical consequences ripple through the rest of this article: sampling decisions are made per segment rather than per task, so a policy that keys on total task cost or total step count has to reconstruct that from the links instead of reading it off a root span, and the diagnostic workflow gains a step, because you now assemble the trajectory across traces before you can read it. None of that is a reason to avoid links; it is a reason to decide early which of the two shapes your agent has.

For attribute names, do not invent a vocabulary. Use the emerging OpenTelemetry GenAI semantic conventions, which cover model calls, agent spans, MCP, and provider-specific details. Two things about them are worth knowing before you build on them. First, they are moving: the conventions were extracted out of the core specification into their own repository and are still evolving — the repository's own schema URL is still marked as a TODO — so pin the version you validate against and treat the schema as something you can migrate, not something you can assume. Second, a rename already happened: gen_ai.system became gen_ai.provider.name in semantic conventions v1.37.0, with the old name deprecated. Copy the deprecated attribute from a blog post and your traces will silently disagree with everyone else's tooling. The other widely used vocabulary is OpenInference, maintained by Arize, which defines a fixed set of span kinds — LLM, CHAIN, TOOL, RETRIEVER, RERANKER, EMBEDDING, PROMPT, AGENT, GUARDRAIL, EVALUATOR — and is what most of the tracing UIs in this space speak natively. Pick one and map the other; a bespoke schema that cannot be joined to either is a dead end.

Instrumenting the trajectory

The shape is ordinary OpenTelemetry — nested spans — with the attribute discipline that makes the spans answer incident questions. The non-obvious parts are the ones people leave out: the tool span records whether the effect succeeded, and the verifier span records a verdict.

Four details in that snippet carry more weight than they look.

The verifier sets the task status. The tool marks itself ERROR when the effect failed; the verifier marks the task ERROR when the output does not satisfy the contract. The second is what turns a silently wrong answer into a signal your existing alerting can consume, and it is the status the tail-sampling policy reads two sections down. Without it, your error rate measures the network, not the product.

Required attributes go on every span, and the tool call id is what makes a trajectory readable. The GenAI conventions mark gen_ai.provider.name as required on agent, inference, and tool spans alike, and gen_ai.tool.call.id as conditionally required on tool spans — the unique identifier of the tool call being executed. The first is easy to set once on the root and then forget; the second is the one people skip and later regret, because a model that calls the same tool twice in one turn emits two spans that differ only by their arguments, and the call id is what lets you line the span up against the model's own output. Note what the verifier span does not carry: it uses our own agent.verifier.* namespace rather than a gen_ai.* name, because the conventions define agent, inference, and tool spans — not your application's verification step. Inventing a gen_ai.operation.name value outside the specification's enumeration is how a schema stops being joinable.

The task carries the correlation ids the rest of the stack needs. agent.task.id plus the trace id is what lets you join the agent trace to the row in your application's database, the ticket in the support system, or the user's session. The most common complaint about agent traces is not that they lack detail — it is that they cannot be joined to anything.

Versions belong on the spans, not in a deploy log. Model id and version, prompt hash, retrieval index version, tool manifest version. When a regression appears three weeks later, the only way to attribute it is to compare traces from before and after a change, and that requires the version to be on the span. A trace without versions is a trace you can read and cannot explain.

Getting it in without a rewrite

The shape above is the target. Getting there in an existing codebase is a matter of picking where the spans come from, and the three options have different ceilings.

Provider auto-instrumentation. Most SDKs and several agent frameworks ship OpenTelemetry instrumentation that emits chat spans with token usage and model ids for free. This is the cheapest starting point and it is worth doing first, because it costs a few lines. Its ceiling is low: you get the model calls and nothing about your domain. No task span, no tool outcome, no verifier verdict — which means no way to tell a failed task from a successful one. It is the floor, not the goal.

An LLM gateway or proxy. Routing model traffic through a gateway that logs requests and responses gives you uniform capture across providers, centralized cost accounting, and redaction in one place rather than in every service. The trade-off is the mirror image: excellent coverage of the model-call layer and no idea what the calls were for. A gateway sees a prompt; it does not know which customer, which task, or whether the effect landed. Use it for the model-call plane and still add your own task spans.

A wrapper at your own boundary. One decorator or context manager around the place where a unit of work starts, emitting the task span and the tool spans it owns, is what actually produces a debuggable trajectory. It is more code than the other two and it is the part that cannot be outsourced, because only your application knows what a task is, what the verifier checks, and which tool call is supposed to be irreversible. The maintenance risk is drift: if the wrapper is optional, half the call sites will skip it and you will have traces that look fine and cover 60 percent of traffic — a failure mode worse than having none, because it is invisible.

The pragmatic order is auto-instrumentation for the model calls, a wrapper for the task and tool spans, and a gateway only if you have several services or several providers to reconcile. What matters is that the span schema is decided once, centrally, rather than re-invented by each team that adds instrumentation.

An honest caveat about the whole of this article: what I have described is a platform, and building it assumes a team that can run a Collector, a payload store with retention and encryption, and its own query tooling. Small teams should not build that, and usually do not need to. Langfuse is open source and self-hostable; Arize Phoenix is built on OpenTelemetry and is the natural landing place if you have already standardised there; Braintrust is a commercial platform whose centre of gravity is evaluation rather than tracing. Buying one of those is usually the right first move — the schema you emit is the thing you own, and the vendor is the replaceable part — provided you check two things before committing. First, whether payloads leave your infrastructure, because that decision is made once and reversed rarely. Second, whether the tool can export in a standard enough form that a migration later is a re-ingest rather than a rewrite. The failure mode to avoid is treating the platform as the control: a dashboard you bought is exactly as much of a gate as a checklist you wrote.

What to capture, and what to refuse

Payloads are the reason agent observability is expensive and the reason it is useful. The resolution is not "capture everything" or "capture nothing" — it is a per-field policy, with the failure path getting more than the success path.

• Capture inline the things you will read on every incident: the model's output, the verifier verdict, the tool name, the token counts, the outcome flags. These are small and high-signal.
• Store by reference the things that are large or sensitive: retrieved chunks, tool arguments, tool results. The span carries an opaque id; the payload lives in an access-controlled store you query when you actually need it. This is the same reference-id pattern the audit trail uses, and for the same reason: a reader of the trace should not automatically be a reader of the data.
• Never capture credentials, tokens, and raw personal data that has no debugging value. If a field is both sensitive and useful, redact at the instrumentation boundary rather than hoping nobody looks. Once a payload is in a trace store, everyone with dashboard access has it.
• Capture everything on failure, and store-by-reference on success. This one asymmetry buys most of the debugging value for a fraction of the storage and a fraction of the exposure, because the traces you need in full are the ones that went wrong.

The last line of that policy is the one teams get wrong. A debug trace store and a compliance audit store look similar and must not be the same system: the debug store wants broad read access, short retention, and the freedom to redact; the audit store wants narrow access, long retention, and evidence that nobody edited it. Build one and try to satisfy both, and you will end up with a store that is too permissive to be evidence and too locked down to be useful for debugging.

Cost and tokens as first-class telemetry

For an agent, cost is not an infrastructure line item; it is a product metric. It is also the symptom that shows up first when a prompt regresses, because a slightly vaguer prompt makes the model take more steps, and more steps multiply both latency and spend. Token counts come free with the model call (gen_ai.usage.input_tokens, gen_ai.usage.output_tokens), and the interesting number is not the total — it is the per-task and per-successful-task figure, computed by rolling the trajectory up.

Two accounting gotchas worth instrumenting around. Retries double-count. If your SDK retries a rate-limited call, the successful retry and the failed attempt are two model calls and two token counts; a naive rollup attributes cost to a task that produced one answer. Streaming and caching move the numbers. A streamed response that the user abandons still burned tokens, and cached prompt tokens are billed differently from fresh ones. Emit the provider's own usage numbers per call rather than estimating from characters, and keep the retry count on the span so the rollup can tell the difference between "expensive task" and "task that fought the rate limiter".

Sampling agents without losing the interesting cases

You cannot keep every trace. Agent traces are long, payload-bearing, and expensive to store, and a head-based sampler — the default, decided at the start of the trace — is the wrong shape for them, because the attributes that make a trace interesting are only known at the end. Whether the task failed, how many steps it took, what it cost, whether the verifier approved: none of that exists when the first span is emitted.

That is what tail sampling is for: buffer the spans, decide at the end. The Collector's tail sampling processor is built for exactly this, and its policy semantics matter when you configure it. Policies are effectively OR'd for keeping — any policy that votes to sample keeps the trace — while a drop policy vetoes it. So you express "keep the interesting ones, sample the rest" as a list of keep policies plus one probabilistic policy.

Four operational facts about this processor that bite in production. All spans of a trace must reach the same Collector instance, because the decision is made from the buffered whole; the documented way to scale is two layers of Collectors, the first using the load-balancing exporter to route by trace id. The buffer is bounded — num_traces keeps traces in memory while waiting, and when it overflows the oldest trace is dropped before it can be sampled, which is the sampling_trace_dropped_too_early metric. Raising num_traces or lowering decision_wait fixes it, both at a cost. Late spans get a second decision unless you configure a decision cache, which produces a trace whose head and tail were sampled by different rules; the decision-cache settings exist for this. And decision_wait has to exceed your task's duration, or long agent runs are decided before they finish — 60 seconds in the example above is a choice about your p99 task length, not a default to copy.

One dependency to keep in mind: these policies evaluate per trace. If your trajectories are segmented and linked rather than nested — the long-running case from the span-model section — then a policy keyed on step count or total cost sees one segment, not the task, and has to have those totals reconstructed from the links before it can decide anything meaningful.

From telemetry to diagnosis

Dashboards are not diagnosis. The workflow you are instrumenting for has a specific shape, and it is worth designing the schema around it:

1. Find a bad task. From an alert, a user report, or a sampled review. The entry point is a task id, so the task id has to be reachable from the systems where complaints arrive.
2. Read the trajectory, top to bottom. The root span gives the outcome and the aggregate cost; the children give the path. You are looking for the first step that looks wrong, not the last one that went wrong — an error on step nine is usually caused by a bad retrieval on step two.
3. Localize to a span class. Was it a retrieval that returned the wrong documents, a model call that ignored the context, a tool that silently wrote the wrong value, or a verifier that approved something it should have rejected? Each of those is a different fix and a different owner.
4. Replay, do not reproduce. Take the frozen inputs — the prompt, the retrieved context, the tool results — and replay the call many times. You are not looking for the same output; you are characterizing the output distribution, and then checking whether that distribution has shifted relative to a baseline. This is the step that requires the payload store and the versions on the spans, and it is the step that people cannot do when they only logged status codes.
5. Confirm on fresh data. A hypothesis from one trajectory is a lead. The confirmation is the change in the aggregate, or in the bucket the task came from.

Everything above depends on two properties that are cheap to add early and painful to retrofit. The span schema has to be stable enough to diff — if attribute names change every sprint, no baseline survives. And the payloads have to be recoverable — a reference id is only useful if the thing it points at was actually stored, with a retention window long enough to cover the incident.

What to alert on

Not CPU. The alertable symptoms of an agent are behavioural, and each one exists because a specific failure mode is otherwise invisible:

• Task success rate, per bucket. The headline. Per bucket, because the aggregate hides the cohort that broke.
• Verifier rejection and abstention rate. The leading indicator. A rising abstention rate is the system telling you its evidence got worse, before users notice.
• Steps per task (p95, and the count above a threshold). Loop detection. An agent that starts taking 20 steps where it took 6 has a prompt, tool, or retrieval regression, and it will show up in cost next.
• Cost per successful task, on its derivative. Alert on the rate of change, not the level, because a step change is what a regression looks like.
• Tool error rate, per tool. Isolates a broken integration from a confused agent.
• Retrieval health. Index freshness and score distributions, sampled. It degrades before anything else and it is the most common root cause.
• Latency per step, not per task. A p95 task latency tells you the trajectory was slow; the span-level view tells you which step.

Two disciplines keep this from becoming noise. Alert on the change in a metric rather than its absolute value, and burn-rate style rather than on totals, because a provider-side model update moves every request in the affected bucket at once. And decide per symptom whether it gates or merely monitors — the gate-vs-monitor split is the same one the evaluation piece argues for, and a gate that fires on noise gets disabled within a week, which protects nothing.

Three data planes, not one

The useful mental model is three separate stores with three different questions, and conflating them is why so many agent observability projects produce a beautiful dashboard nobody debugs with.

• Traces answer what happened. Spans, attributes, outcomes, cost. Short retention, broad access, optimized for reading one trajectory.
• Evaluations answer was it good. Graded samples, judge verdicts, human labels, regression baselines. Longer retention, per-dimension metrics, optimized for trends.
• Effects answer what changed in the world. The row that was written, the email that was sent, the refund that was issued. This plane does not live in your LLM telemetry at all — it lives in the systems the agent acted on, and it is the one that matters when the question is "what did it actually do".

The third plane is what distinguishes agents from chatbots, and it is the one teams forget to join up. A trajectory can look perfect — correct tool, correct arguments, verifier approval — and still have written to the wrong record, because the identifier it was given was wrong. Only the target system knows that. The join key is the task id on the span and the same id recorded on the effect, which is a one-line change in the tool implementation and a permanent capability afterwards.

Failure modes of agent observability

Instrumentation has its own failure modes, and they are as expensive as the ones it is meant to catch.

• Instrumenting the wrong layer. Beautiful HTTP metrics and no trajectory. The dashboard is green during every incident.
• Logging everything. Prompts and payloads inlined on every span, retention set to forever, costs climbing, and nobody able to find the trace they need because the store is a landfill.
• No versions on spans. You can read the trajectory and cannot explain it, so nothing is attributable to a change.
• Head sampling. The interesting traces are, by construction, the ones you decided not to keep at the moment when you knew nothing.
• PII in the trace store. Broad access plus captured payloads equals a data-protection incident with a slow fuse.
• Unreplayable traces. The reference ids point at payloads that were never stored, or were deleted 24 hours before the incident was reported.
• Coarse spans. One span per task and nothing inside it. You can see that it failed and not which step, which is the same as having no telemetry for diagnosis.
• Conflating traces with evals. "We have tracing" used as evidence that quality is measured. One tells you what happened; the other tells you whether it was good.
• Observability as compliance theatre. A log retained because a policy mentions logs, with no query anyone runs. An audit trail is not a debugging instrument.

When this does not fit

Building this pipeline is real work, and for some stages of a product it is the wrong work.

Before the task is defined, there is no stable outcome to key on: you do not yet know what "success" means, so the verifier span has nothing to assert, and the sampling policies have nothing to filter on. Instrument the minimum — inputs, outputs, model version, tool calls — and spend the effort on the product instead. The trajectory schema can come later; the payloads cannot be reconstructed, which is why they are the part worth capturing from the beginning.

At low volume with a human reviewing everything, a dashboard is a worse tool than reading the cases. If ten tasks a day go through a reviewer, the reviewer is the observability system, and the instrumentation you need is the one that puts the trajectory in front of them, not the one that renders percentile charts.

When you have no failure to debug yet, the honest move is to instrument the floor and wait. Spans, task ids, versions, tool outcomes, token counts — cheap, always useful. Tail sampling policies, payload stores with retention tiers, per-bucket alerting: build them when a real incident tells you what you wish you had, because the policies you guess at today will be wrong in a way you only discover during the incident you built them for.

And there is a floor that holds even in the smallest system, covered at length in the companion article on hallucinations and context problems: log the inputs, the outputs, the model and prompt versions, and the tool calls, and put a deterministic check on anything that writes state. That costs almost nothing, it is what makes real observability possible later, and skipping it is the one shortcut that cannot be undone.

Summary

Agent observability is not classical observability with extra attributes. Three assumptions invert: you cannot reproduce a run, so you replay frozen inputs and characterize distributions; failure is not an exception, so a verifier has to manufacture the signal; and the unit is a trajectory, not a request, so the root span is the task.

The instrumentation follows from that. Span the task, then each model call, tool execution, retrieval, and verification, with the effect outcome and the versions on the spans and a task id that joins to the rest of the stack. Use the OpenTelemetry GenAI conventions rather than a bespoke vocabulary, pinned to a version, and know that gen_ai.provider.name replaced gen_ai.system. Capture payloads per field, store the large and sensitive ones by reference, capture everything on failure and pointers on success, and keep the debug store strictly separate from the audit store. Compute cost per successful task, alert on behavioural symptoms and on rates of change, and sample from the tail so the interesting traces survive a decision made with full knowledge. Then keep the third plane in view: the trace tells you what the agent decided, and only the systems it touched can tell you what it did.

The test of the whole thing is not the dashboard. It is whether, when a customer says the refund went to the wrong order, you can find that task in under a minute, read the trajectory, point at the step that went wrong, and replay it — and whether that answer took an afternoon or a week.