AI Agent Observability: The Builder's Setup Guide (OTel, Traces, Spend)

If you ship agents, ai agent observability is how you see every step they take, plus the cost, latency, and quality on that step. Owners ask for ai agent observability after the first silent failure, not before. A tool returns stale data. A planner loops. Memory serves the wrong file. The reply still reads fine.

That is worse than an outage. Nobody gets paged. You hear about it from the client, usually after the damage is done.

According to LangChain's State of Agent Engineering report, 89% of organizations said they have implemented some form of agent observability. The same report says 62% have detailed step-level tracing. A checkbox is not a system you can run a client on.

A KPMG survey, cited by IBM, found 88% of organizations exploring or actively piloting AI agent initiatives. Gartner predicts that by 2028 more than a third of enterprise software applications will include agentic AI. Volume is coming. Blind volume is expensive.

This guide stays vendor-neutral. Instrument once. See traces, spend, latency, and quality in one place. Know what to alert on. Attribute spend per client and per agent. If you have not mapped where agents belong in the operation yet, start with the AI Diagnostic.

Why do agents fail silently?

Agents do not crash like a web app with a red status page. They degrade. The user still gets a paragraph. It just happens to be wrong.

A tool call returns yesterday's balance. A planner takes the same step twice. A memory store hands back another client's file. The sentence structure is clean, so a busy operator skims it and moves on.

Dynatrace lists the risks of unobservable agents as business impact, operational impact, compliance impact, and cost impact. Incorrect responses drain revenue and damage customer trust. You usually find out from the client, not from a dashboard.

Confident AI walks through the shape of that failure. Their example agent, Garry, refunded invoice #4417 instead of #4471. It hallucinated a transposed ID that belonged to a different customer. The action looked complete. It was the wrong action.

On τ-bench, a benchmark of tool-using customer-service agents, even state-of-the-art function-calling models complete fewer than half of real tasks. Confident AI says tool-use errors, like calling the wrong API or passing the wrong parameters, are the primary failure mode on that benchmark.

A wrong answer that reads well burns trust and budget at the same time. That is the failure this setup is built to catch. Uptime can stay green while the work is bad.

What does AI agent observability actually mean?

AI agent observability is the practice of capturing and inspecting every step an agent takes (model calls, tool calls, retrievals, memory reads, and decisions) along with the cost, latency, and quality signals on each step.

IBM defines it as the process of monitoring and understanding the end-to-end behaviors of an agentic ecosystem, including interactions with large language models and external tools. Braintrust puts the capture list in operator language: tool selection, tool arguments, model responses, memory reads, memory writes, state transitions, and decision branches.

Traditional observability assumes a path you designed. LangChain says traditional observability tools capture request-response cycles but fall short for AI agents. Dynatrace says observability must evolve from describing what happened to explaining why it happened. The new data points, which do not exist in a normal service, include prompts, reasoning chains, contextual inputs, tool calls, multi-agent interactions, and outputs.

Logs tell you something happened. Traces tell you why the agent chose it. IBM groups the raw telemetry as MELT data: metrics, events, logs, and traces. For an agent, the trace is the only record of a path the system picked at runtime.

No logo on the dashboard changes that job. The record has to survive a backend swap. Where those agents sit in a client operation is laid out on how it works.

What should you instrument: traces, spans, tokens, cost, and evaluations?

The unit of observability is the span. Confident AI's framing is the one to keep: spans (LLM, tool, retriever, agent) make up traces, and traces group into threads. That is agent tracing you can actually debug.

One trace per user task. One span per model call, tool call, retrieval, and memory operation. Parent-child structure is what makes a multi-step bug readable. If the parent has no children, you have a log line with a costume on.

Braintrust names four pillars of an agent trace: tool calls, reasoning steps, state transitions, and memory operations. Instrument the non-model pieces. Memory stores, retrievers, and planners fail quietly more often than the model does. For example, a refund agent might look fine on the model span and still pull the wrong invoice from a stale retriever.

On every model span, capture input tokens, output tokens, cached tokens, and the model name. Cost can then roll up without a spreadsheet reconstruction. IBM's cost pattern is the one to copy: if certain customer questions use 10 times more tokens than others, redesign how the agent handles those requests.

Latency needs two cuts. Time to first token, and total span duration, split by model call versus tool call. Owners often blame the model. The trace often shows a slow tool, a retry loop, or a retrieval that ran twice.

Attach a quality score to the trace. Do not park agent evaluation in a side file nobody opens on Friday. Online evals score live traffic. Offline evals score a fixed set you can rerun after a prompt or tool change. Confident AI's line is the standard: observability without evaluation is just expensive logging.

If those agents are doing client work as digital employees, the score belongs on the same trace as the tokens. Otherwise you cannot tell a cheap failure from an expensive success.

How do OpenTelemetry GenAI semantics work in practice?

OpenTelemetry matters because it is an open standard. You are not locked to one vendor SDK. You can swap backends without rewriting the instrumentation.

The GenAI observability project inside OpenTelemetry is actively defining semantic conventions to standardize AI agent observability. The initial AI agent semantic convention is based on Google's AI agent white paper. Experimental conventions already exist for models at the GenAI semantic convention. Use those attribute names even while the agent conventions are still settling. Model name, token counts, and operation type should not be free-text inventions on every team.

In practice, give model calls and agent operations their own span kinds. Put standard attributes on them: model name, input tokens, output tokens, operation type, and a stable tool name. Dynatrace describes telemetry standardized via OpenTelemetry and OpenLLMetry, then unified into one correlated model. Treat that as a pattern. Do not treat it as a product pick.

LangChain notes that during prototyping, print statements are sufficient. The tipping point is narrower than most owners think: the agent uses two or more tools in sequence, you are past a few dozen requests a day, or someone other than the original developer has to debug a failure. Client work hits that line fast.

Move in stages. Do not boil the agent on day one.

1. Wrap the model call. Record model name, token counts, latency, and whether the call was cached.
2. Wrap each tool call, retrieval, and memory read or write as its own child span, with arguments, return status, and duration.
3. Wrap the full agent loop as the parent trace. Set client ID and agent ID at the entry point, before the first model call.

Some frameworks bake instrumentation in. OpenTelemetry notes that CrewAI follows that pattern, and warns that baked-in instrumentation risks version lock-in if the framework's OpenTelemetry dependencies lag behind upstream updates. Braintrust says most production teams mix frameworks: OpenAI Agents SDK, LangGraph, Mastra, Pydantic AI, and custom in-house orchestrators. For unsupported frameworks, OpenTelemetry instrumentation with the OpenTelemetry SDK is the fallback path. OpenTelemetry also lists IBM Bee AI, IBM wxFlow, CrewAI, AutoGen, Semantic Kernel, LangGraph, and PydanticAI in this space. IBM names BeeAI, LangChain, LangGraph, and AutoGen as open-source options teams use to build multi-agent systems.

Opinion: use baked-in tracing only if you are sure you will stay on that framework. If you expect to change backends, opentelemetry for agents is the cleaner default. Portability is the point. A faster first week is not.

What minimal stack can you stand up?

The smallest setup that holds is an OpenTelemetry collector, a trace backend, a metrics store, and a dashboard. Nothing exotic. If the stack needs a specialist just to see one failed task, it is too big for the first deployment.

You will not keep every trace at volume. LangChain says production scale beyond 1,000 daily runs overwhelms human capacity to review every trace by hand. Sample hard on clean successes. Keep every failure, every eval miss, and every high-cost trace. Set retention by tier: short for routine success, longer for anything a client might dispute.

Open source gives control and a lower license cost. It also gives you the ops load. Commercial ai agent observability tools give speed and support, and a meter that scales with seats or spans. Read the meter before you sign.

Datadog says it bills only per LLM span, defined as a single call to an LLM provider, and that tool, workflow, agent, embedding, and retrieval spans are free. That is one vendor's meter, not a recommendation. Braintrust publishes a free tier of 1 GB of processed data and 10k evaluation scores per month. Same rule. A price-page fact is not a shortlist.

Braintrust notes that Datadog and New Relic capture infrastructure signals like request rate, latency percentiles, and span counts. Useful. Not the whole picture. You still need the agent path, the tokens, and the quality score next to those percentiles.

Skip fake precision on cost benchmarks. Roll spend up by task type and compare it to the fee you charge for that task. If trace cost eats the margin, the workflow is the problem, not the chart. A working session on the wiring sits in the workshop. The operating layer around the agents is the AIOS.

How do you attribute spend per client and per agent?

Tag every span with client ID and agent ID at the entry point. Do it there. A cleanup job three months later will miss the tasks you most need to explain.

Roll spend three ways: by client, by agent, and by task type. That rollup is what turns a trace store into a pricing and margin decision. A single blended average hides the client who is quietly expensive.

Then tie trace cost and quality to the outcome the agent produced. A cheap trace that failed the task is not a win. An expensive trace that finished the work might be. You want both numbers on the same row, or you will cut the wrong thing.

This is the owner question. Is the agent earning its keep, or quietly draining it?

For example, a refund agent might show a low average cost and still lose money if one slice of tasks uses far more tokens and fails more often. IBM's pattern is the tell: certain questions use 10 times more tokens than others, and the fix is to redesign how the agent handles that slice. Split it out. Cap it, reprice it, or change the tool path.

Same tags are what finance will actually read. If you sell the work, per-client spend is how you defend the invoice without a forensic project. Where the work shows up by sector is on industries.

What should you alert on, and where do guardrails sit?

Alert on few things. Alert only on things a human will act on. An alert nobody acts on is noise, and noise is how real failures get ignored.

Signal	Page when	Why it matters
Cost per task	Above the cap you set for that task type	Margin leak before the invoice
Latency	A regression versus the last stable release	A slow tool hiding behind a fine model
Tool failure rate	Retries or errors climbing	The common break on tool-using work
Eval score	Below the floor you set for that agent	Quality dropping while uptime stays green
Loop detection	The same tool or plan step repeating	Spend with no progress
Human escalation	More handoffs than your baseline	A business signal, not just an infra one

Guardrails are the runtime counterpart. Hard spend caps. Tool allowlists. Output checks that stop a bad action before it ships. Dynatrace describes behavioral monitoring that identifies unsafe actions, hallucinations, or deviations from policy across agent frameworks. That is the shape of the check, whoever implements it.

Datadog lists controls buyers ask for: precise alerting, role-based access control, sensitive data protection, and HIPAA compliance. Use that list as a checklist for any stack. Do not use it as a reason to crown one vendor.

LangChain describes AI-assisted debugging as letting you ask "why did the agent do this?" in natural language and get answers in seconds. Useful after the alert fires. Not a substitute for the alert. Someone still has to own the page.

If you are buying agents rather than building them, ask for this view before go-live. Traces, spend by client, a quality score, and a short alert list. If the seller cannot show it, you are buying a demo. The path from a staffed agent into daily work is on digital employees.