Tracing AI Agents in Production with OpenTelemetry: What to Capture

An AI agent that fails in production rarely throws an exception. It picks the wrong tool, loops four times instead of two, or answers confidently from a stale lookup. Your HTTP logs show a clean 200. The trouble is in the middle, and you cannot see the middle unless you recorded it.

This guide covers how to record it using OpenTelemetry's GenAI semantic conventions: which spans to emit, which attributes earn their storage cost, what to keep out of your traces on purpose, and the one caveat that should shape how you adopt any of this, which is that the conventions are still under development and have recently moved.

Why agents need traces, not just logs

A single user request to an agent fans out into several model calls and tool calls, and the number is not fixed. The model decides at run time. Two identical requests can take different paths, cost different amounts, and fail in different places.

A trace is the right shape for that. One root span covers the whole agent run. Child spans cover each model call and each tool call, with timing and status on every one. When a run is slow, you can see whether the time went into the model, into a tool, or into the loop count. When a run is wrong, you can see which step diverged.

If you are still deciding where agents belong in your business at all, our guide to AI agents for business covers that first. This article assumes you already have one running, or are about to.

The span tree to emit

OpenTelemetry's GenAI conventions describe operations by name. In the walkthrough Microsoft's James Newton-King published on the OpenTelemetry blog, the agent trace has three span names: invoke_agent for the top-level agent run, chat for each LLM call, and execute_tool for each tool invocation. Nest them the way the work actually nests.

The invoke_agent span's kind has changed in the spec history. According to the semantic-conventions changelog, version 1.38.0 added client versus internal span-kind guidance for invoke_agent, and version 1.41.0 split it into separate client and internal spans. In practice: an agent running inside your own process is an internal span; a call out to a remotely hosted agent service is a client span. Check the current text before you pick one.

Attributes worth recording

The Microsoft walkthrough lists the attributes that matter most for debugging. These are the ones I would start with:

• gen_ai.operation.name on every span, set to the operation (invoke_agent, chat, execute_tool).
• gen_ai.provider.name and gen_ai.request.model, so you can slice latency and errors by model. The provider attribute was previously called gen_ai.system; the changelog records the rename in 1.37.0, so older tutorials will show the old key.
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on each model call. Cost per run is the sum over the tree.
• gen_ai.response.finish_reasons, for example stop or tool_calls. A run that keeps ending in tool_calls is a loop you want to find.
• gen_ai.conversation.id, to join several runs belonging to one conversation, and gen_ai.agent.name and gen_ai.tool.name to know who did what.

Then add your own, namespaced so they cannot collide with the spec. Step count, the configured step limit, and a boolean for whether a tool succeeded are cheap and tell you more than most standard attributes do.

What to leave out of your traces

By default the GenAI instrumentation shown in that same walkthrough does not capture prompt content or tool arguments, because they can contain sensitive data. Turning content capture on adds system prompts, full messages, tool arguments and tool results to span attributes (gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages). The post also warns those payloads can be large.

That is a good default and you should keep it for production. Customer messages, order details and internal documents will otherwise land in whatever backend stores your traces, with that backend's retention and access rules rather than your application's. If you enable content capture for debugging, do it for a short window, on a sampled subset, with redaction before export, and with the trace backend covered by your data-protection review. If your agents handle personal data of people in the EU, see our piece on what applies to your chatbot under the EU AI Act for the transparency side of the same question.

A minimal instrumented agent

This TypeScript sketch uses only the stable @opentelemetry/api package (we type-checked it against version 1.9.1 under strict mode; we did not run it against a live model or collector). It emits the three span names above, records token usage and finish reasons, and deliberately does not attach tool arguments or results. Wire up an SDK and exporter separately, and replace the stubbed model call with your provider SDK.

import { trace, SpanKind, SpanStatusCode, type Span } from "@opentelemetry/api";

const tracer = trace.getTracer("support-triage-agent");

type ToolResult = { ok: boolean; body: string };

async function callModel(prompt: string): Promise<{ text: string; inTokens: number; outTokens: number; finish: string }> {
  return tracer.startActiveSpan("chat", { kind: SpanKind.CLIENT }, async (span: Span) => {
    try {
      span.setAttributes({
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "openai", // use the value your provider's convention defines
        "gen_ai.request.model": "your-model-id",
      });
      const res = { text: "stub", inTokens: prompt.length, outTokens: 4, finish: "stop" }; // replace with your SDK call
      span.setAttributes({
        "gen_ai.usage.input_tokens": res.inTokens,
        "gen_ai.usage.output_tokens": res.outTokens,
        "gen_ai.response.finish_reasons": [res.finish],
      });
      return res;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

async function runTool(name: string, args: unknown): Promise<ToolResult> {
  return tracer.startActiveSpan("execute_tool", async (span: Span) => {
    try {
      span.setAttributes({ "gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": name });
      // Do NOT attach args or results here unless content capture is explicitly enabled and redacted.
      const result: ToolResult = { ok: true, body: JSON.stringify(args) };
      span.setAttribute("app.tool.ok", result.ok); // custom attribute: namespace your own keys
      return result;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

export async function invokeAgent(task: string, conversationId: string) {
  return tracer.startActiveSpan("invoke_agent", { kind: SpanKind.INTERNAL }, async (span: Span) => {
    try {
      span.setAttributes({
        "gen_ai.operation.name": "invoke_agent",
        "gen_ai.agent.name": "support-triage",
        "gen_ai.conversation.id": conversationId,
        "app.agent.max_steps": 6,
      });
      let steps = 0;
      let answer = "";
      while (steps < 6) {
        steps++;
        const reply = await callModel(task);
        if (reply.finish !== "tool_calls") { answer = reply.text; break; }
        await runTool("lookup_order", { id: "redacted" });
      }
      span.setAttribute("app.agent.steps", steps);
      return answer;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

Two details matter. Every span is ended in a finally block, because a leaked span corrupts the whole tree. And errors are recorded on the span that failed, not only on the root, so the trace points at the broken step.

Retrieval steps deserve a span too. If your agent searches documents before answering, the question of whether it retrieved the wrong thing or reasoned badly from the right thing is usually the first one you will ask. Our RAG versus long-context decision guide covers how to choose that architecture; whichever you pick, trace it.

If your agents call tools over MCP, the protocol's own changes affect where state lives and how requests are retried, which changes what your traces show. See our MCP 2026-07-28 migration guide.

The caveat: the spec is still moving

Treat the conventions as a moving target. The OpenTelemetry blog post describes them as already in use and under active development, not as stable. The main semantic-conventions changelog shows the GenAI definitions were deprecated in that repository in version 1.42.0 and moved to a dedicated repository, open-telemetry/semantic-conventions-genai. The old pages on opentelemetry.io now show a notice pointing there, and the attribute registry on the main site marks the gen_ai.* entries as deprecated and moved. The new repository's README does not state a stability level that we could find.

What that means for you:

1. Pin the version of any instrumentation library you use, and read its release notes before upgrading. Attribute names have changed before (see the rename above).
2. Keep your own dashboards and alerts behind a thin layer, such as a handful of recording rules or saved queries, so a rename is a one-place fix.
3. Prefer your own namespaced attributes for anything business-critical, like cost per task, and treat gen_ai.* as the portable part.
4. Check the current text in the new repository before copying attribute names from any article, including this one.

Tool support also varies. The same walkthrough notes that trace support in some coding assistants is still labelled beta.

A rollout order that works

Start with one agent and the three span names. Add token usage, then finish reasons, then your own step counters. Build two views before anything fancy: slowest runs by total duration, and runs by number of model calls. Those two catch most loop and latency problems.

Only after that, add evaluation: sample traces, label a few dozen by hand, and use them to decide what you automate. Traces are the raw material for evals, so having them first is not wasted effort.

Where this goes wrong

• Span explosion. Streaming every token chunk as its own span makes traces unreadable and expensive. Record chunk timing as metrics, not spans.
• Unbounded content. One long document in an attribute can exceed backend limits and get dropped silently.
• No sampling plan. Tracing everything is affordable in a pilot and expensive at volume. Decide early to keep all errors and slow runs and sample the rest.
• Trusting the model's own account. The trace records what the system did. An agent's explanation of what it did is not evidence.

Getting help

Agent observability is easy to bolt on late and painful to retrofit. If you are building agents and want the tracing, evaluation and access controls designed in from the start, our AI automation service covers agent builds, and our consulting team can review an existing design. You can also book a call to talk through your setup.

Sources

• OpenTelemetry blog, James Newton-King: "Inside the LLM Call: GenAI Observability with OpenTelemetry" (published 14 May 2026): https://opentelemetry.io/blog/2026/genai-observability/
• OpenTelemetry semantic-conventions changelog (v1.33.0, 1.37.0, 1.38.0, 1.41.0, 1.42.0 entries): https://github.com/open-telemetry/semantic-conventions/blob/main/CHANGELOG.md
• OpenTelemetry GenAI semantic conventions repository: https://github.com/open-telemetry/semantic-conventions-genai
• Notice on the old GenAI page: https://opentelemetry.io/docs/specs/semconv/gen-ai/