The complete LLM tracing guide (2026)

Why logs fall short for LLM apps

When a traditional web service misbehaves, a log line usually tells you enough. The same input produces the same output, the code follows a known path, and a 200 response is a strong signal that the request worked. LLM applications break all three assumptions, which is where logging starts to fail.

Picture a support assistant built on retrieval. A customer asks why they were charged twice last week. The assistant embeds the question, runs a vector search over your help docs, retrieves a few passages, calls an order-lookup tool to fetch the customer's recent transactions, and sends the assembled context to a model to generate the final reply. The reply comes back wrong. It tells the customer there was no duplicate charge, even though there clearly was one.

Now look at what the log shows. It records the final string the model produced and little about the chain that produced it. You cannot see which passages the retriever returned, whether the order-lookup tool fetched stale data, what the assembled prompt contained, or how many tokens each step used. The failed answer is visible, but the evidence behind it is missing.

Logs show the final outcome, but tracing exposes the request path behind it. When retrieval, tool calls, and model steps are visible in one trace, teams can see which step introduced incomplete or incorrect context.

Two properties of LLM apps make log-only debugging unreliable. First, LLM requests are non-deterministic, so the same question can take a different path across runs, and a failure may not reproduce from the input alone. Second, modern requests often move through retrieval, tool calls, and multiple model hops before returning a response. A problem in any step can corrupt the final answer even when no system error occurs.

Logs were built for discrete events, while an LLM request is a connected chain of decisions, context, and generated output. Debugging that chain requires a structured record of the request itself, and that record is an LLM trace.

What is an LLM trace

An LLM trace is the complete, structured record of one request as it moves through an application. Instead of treating the request as a single black box, tracing breaks the request into the steps that actually ran. Each step becomes a span, and those spans connect into a tree that mirrors the execution path.

In the support assistant example, the embedding step, vector search, order lookup tool call, and final model call each become a separate span within a single trace. That structure lets you inspect the request as it happened, from the initial user question to the final response.

Spans and the trace tree

A span represents a single operation, with a start time, an end time, and the data produced by that operation. Spans are arranged in a parent-child structure, so the top-level request span becomes the parent, and the retrieval, tool, and model spans sit underneath it. When one step calls another, the child span remains nested inside the step that triggered it.

The trace tree is what makes complex runs easier to read later. You can open the parent span for the duplicate-charge question, expand the retrieval span to see which passages came back, then open the tool span to see what the order lookup returned. The request remains inspectable in execution order, even when the application uses agents, tools, retries, or multiple model calls.

What each span records

The value of a trace depends on the data each span stores. A well-formed span captures the inputs and outputs of the step, timing, model parameters, errors, retries, token usage, latency, cost, and identifying metadata such as user or session ID.

In the support assistant example, each span answers a different debugging question:

Retrieval span: Records the embedded query, the passages returned by vector search, and how much retrieved text was passed into the next step. If the retriever pulled the wrong help article, the retrieval span shows the mismatch.

Tool span: Records the tool name, arguments, raw return value, duration, errors, and retries. If the order-lookup tool queried the wrong date range or returned stale transaction data, the tool span contains the evidence.

Model span: Records the assembled prompt, model, parameters, completion, token counts, and latency. Token-level tracing captures prompt, cache, completion, and reasoning tokens for each model call. Cost can then be estimated from those token counts and the provider's pricing.

LLM spans often contain full prompts, retrieved documents, tool outputs, and model responses, making them much larger than the small payloads traditional distributed tracing systems were designed to store. Braintrust uses Brainstore, its database for complex AI traces, to support fast ingestion, search, and inspection across large nested trace data.

LLM tracing vs. logging

The table below shows where logging ends and LLM tracing becomes necessary.

How to add LLM tracing

Adding LLM tracing means deciding which parts of the request the SDK can automatically capture and which application-specific steps require manual spans. Most teams start with automatic capture for model calls, then add manual spans for retrieval, tools, business logic, and agent steps.

Auto-instrumentation vs manual spans

Auto-instrumentation is the fastest way to start and the recommended default. One call at startup patches supported AI libraries, so LLM calls are captured with inputs, outputs, latency, token usage, and cost without wrapping each client individually. In Python, that call is auto_instrument().

import os
import braintrust

# Call once at startup — all LLM calls are traced automatically
braintrust.auto_instrument()
braintrust.init_logger(
    api_key=os.environ["BRAINTRUST_API_KEY"],
    project="My Project (Python)",
)

from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.responses.create(
    model="gpt-5-mini",
    input="What is the capital of France?",
)

Manual instrumentation captures the parts of the request that are specific to your application, such as retrieval, tool functions, and business logic. In Python, you can decorate a function so its inputs and outputs are logged, and nested calls become nested spans.

from braintrust import init_logger, traced

logger = init_logger(project="My Project")

# Decorate a function to trace it automatically
@traced
def fetch_user_data(user_id: str):
    # This function's input (user_id) and output (return value) are logged
    response = requests.get(f"/api/users/{user_id}")
    return response.json()

# Use the function normally
user_data = fetch_user_data("user-123")

SDK wrapping in Python and TypeScript

SDK wrapping gives you explicit control over which clients are traced. Use this path when you do not want to patch every supported library at startup, or when you want tracing applied only to specific model clients.

initLogger({
  apiKey: process.env.BRAINTRUST_API_KEY,
  projectName: "My Project (TypeScript)",
});

// Wrap the OpenAI client to trace all calls
const client = wrapOpenAI(new OpenAI({ apiKey: process.env.OPENAI_API_KEY }));
const response = await client.responses.create({
  model: "gpt-5-mini",
  input: "What is the capital of France?",
});

The Python equivalent uses wrap_openai on the client.

import os
import braintrust
from braintrust import wrap_openai
from openai import OpenAI

braintrust.init_logger(
    api_key=os.environ["BRAINTRUST_API_KEY"],
    project="My Project (Python)",
)

# Wrap the OpenAI client to trace all calls
client = wrap_openai(OpenAI(api_key=os.environ["OPENAI_API_KEY"]))
response = client.responses.create(
    model="gpt-5-mini",
    input="What is the capital of France?",
)

Using OpenTelemetry

Teams that already use OpenTelemetry can send GenAI spans to Braintrust without replacing existing instrumentation. Braintrust implements the OpenTelemetry GenAI semantic conventions, so spans with gen_ai attributes are mapped to structured inputs, outputs, metadata, and token metrics. The span processor drops into the existing tracer provider.

import os

from braintrust.otel import BraintrustSpanProcessor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

# Configure the global OTel tracer provider
provider = TracerProvider()
trace.set_tracer_provider(provider)

# Send spans to Braintrust.
provider.add_span_processor(BraintrustSpanProcessor())

The TypeScript path uses the same processor as the Node SDK.

const sdk = new NodeSDK({
  serviceName: "my-service",
  spanProcessor: new BraintrustSpanProcessor({
    parent: "project_name:your-project-name",
  }),
});

sdk.start();

Teams that already export OpenTelemetry data to another backend can add Braintrust as an additional destination without changing their existing instrumentation.

Framework integrations

Native integrations capture framework-level agent steps as structured spans. Braintrust supports SDK wrappers for the OpenAI Agents SDK, LangGraph, Mastra, Pydantic AI, LangChain, CrewAI, and the Vercel AI SDK, so teams can trace agent runs without manually instrumenting every framework step.

For LangGraph, you register a callback handler, and each graph node becomes a nested span.

import braintrust
from braintrust.integrations.langchain import BraintrustCallbackHandler, set_global_handler
from langgraph.graph import StateGraph

braintrust.init_logger(project="My Project")

handler = BraintrustCallbackHandler()
set_global_handler(handler)

graph = StateGraph(AgentState)
graph.add_node("plan", plan_node)
graph.add_node("act", act_node)
graph.add_edge("plan", "act")

app = graph.compile()
result = app.invoke({"input": "Refund the duplicate charge from last week"})

Each LangGraph node becomes a nested span within the parent agent run, so the plan and act steps remain linked throughout the full execution.

For Mastra, which is a TypeScript-first stack, you register an exporter on the observability config.

const logger = initLogger({ projectName: "mastra-demo" });
const exporter = new BraintrustExporter({
  braintrustLogger: logger,
});
const mastra = new Mastra({
  agents: {
    assistant: new Agent({
      name: "Assistant",
      instructions: "You only respond in haikus.",
      model: "openai/gpt-5-mini",
    }),
  },
  observability: new Observability({
    configs: {
      braintrust: {
        serviceName: "demo",
        exporters: [exporter],
      },
    },
  }),
});

async function main() {
  const agent = mastra.getAgent("assistant");
  const response = await agent.generate("Tell me about recursion in programming.");
  console.log(response.text);
}
main();

For the OpenAI Agents SDK, auto-instrumentation is the recommended path. The same auto_instrument() call captures agent steps, LLM calls, and tool calls as nested spans, so teams do not need a separate code path for that framework.

Tracing overhead, async, and sampling

LLM tracing adds some recording work, but the implementation should keep that work away from the user-facing request path. The main controls are asynchronous logging, streaming behavior, and selective span capture.

Async logging: Trace data is sent in the background instead of blocking the user's request. Background batching is controlled by an async flush setting, enabled by default in the Python and TypeScript SDKs, so clients are not blocked if the logging backend is temporarily unavailable. Serverless environments need extra care because the process can stop as soon as the request completes, so you should flush manually before shutdown.

Streaming: Streamed responses do not need to be logged chunk-by-chunk. Chunks are collected and recorded as a single span, keeping streaming traces readable and avoiding unnecessary logging overhead.

Sampling and filtering: High-volume applications do not always need to send every span. The BraintrustSpanProcessor supports a filter option for sending only AI-related spans and a custom filter hook for finer-grained sampling. Filtering reduces trace volume, but it can also remove surrounding context, so tune sampling based on the spans your team actually needs for debugging, scoring, and monitoring.

Reading and debugging an LLM trace

Capturing traces only helps if engineers can read them quickly. The goal is to move through the request in execution order, inspect the evidence attached to each span, and identify where the final response changed.

Reading a trace is a structured debugging process. Start with the failed output, inspect the retrieved context and tool data behind it, then compare the corrected run to confirm the issue was fixed.

Take the support assistant who wrongly told the customer there was no duplicate charge. You start at the top of the trace tree, which is the span for the whole request, and read the final output to confirm the failure. From there, you move through the child spans in execution order instead of guessing from scattered log lines.

The retrieval span shows which passages were returned and whether the correct help article was included.

The tool span shows the order-lookup arguments and the raw value returned, which is where stale data, an empty transaction list, or the wrong date range would appear.

The model span shows the assembled prompt, so you can confirm whether the duplicate-charge record was available to the model before it generated the answer.

In this example, the tool span is the source of the failure. The order lookup returned an empty list because it queried the wrong date range, so the prompt contained no record of the second charge. The model answered from an incomplete context, and the trace makes that root cause visible without turning the investigation into guesswork.

Comparing runs extends the same workflow. When you change a prompt, fix a tool call, or swap a retriever, compare the new trace against the old one, span by span. The comparison shows whether the fix held, whether downstream behavior changed, and whether latency or cost increased.

From traces to evals

Production traces become more valuable when they feed the same evaluation process that controls future releases. The request data engineers use for debugging can also support online scoring, monitoring, and regression evaluations because the inputs, outputs, intermediate steps, latency, and cost are already structured.

Braintrust lets teams attach scorers to live production traces so quality is measured as requests run, not only after users report a bad answer. Online scoring turns production behavior into a continuous signal that engineering, product, and domain reviewers can use to track failures and prioritize fixes.

curl https://api.braintrust.dev/v1/project_score \
  -H "Authorization: Bearer $BRAINTRUST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "<project_id>",
    "name": "Production scoring rule",
    "description": "Score production traces",
    "score_type": "online",
    "config": {
      "online": {
        "sampling_rate": 1,
        "scorers": [{ "type": "function", "id": "<scorer_function_id>" }],
        "apply_to_root_span": true
      }
    }
  }'

Monitoring uses the same span data that tracing already captures. Latency and cost can roll up into dashboards and alerts, while scorer results indicate whether output quality holds across real traffic. That gives teams a single production view of system behavior and response quality.

Regression testing completes the trace-to-eval workflow. Production traces that fail an online scorer can become eval cases, so the eval suite grows from real user behavior. A failing trace, such as the order-lookup bug, becomes a permanent test case, reducing the risk of the same issue reaching production again.

Braintrust's free tier includes 1GB processed data and 10K evaluation scores per month, so teams can set up tracing, online scoring, and evals before committing to a paid plan.

Start free with Braintrust and instrument your first request →

FAQs: the complete LLM tracing guide (2026)

What is the difference between LLM tracing and logging?

Logging records individual events and answers what happened, while LLM tracing records the full request path and answers why a response came out the way it did. Reach for tracing when the answer depends on retrieval, tool calls, and prompt assembly that no single log line can explain.

Does LLM tracing add latency?

Well-designed tracing keeps the recording work off the user-facing request path, so responses are not blocked. Async logging, batching, and sampling move trace data to the background, though serverless functions need a manual flush before the process exits.

What is a span in LLM tracing?

A span is the record of one operation inside an LLM request, such as a retrieval step, tool call, model call, or response-generation step. Each span provides engineers with a focused place to inspect inputs, outputs, timing, errors, token usage, and the metadata needed to understand the operation.

Should I use OpenTelemetry or an LLM tracing SDK?

Use an LLM tracing SDK for the fastest way to capture model calls and AI-specific metadata. OpenTelemetry is a good fit for teams that already emit OTel spans and want to route GenAI telemetry into their tracing workflow without replacing existing instrumentation.

Can LLM tracing support multi-agent applications?

LLM tracing can support multi-agent applications when agent steps, tool calls, sub-agent calls, and model calls are captured as nested spans. That structure keeps the run inspectable as one execution flow, even when the application branches across several agents or tools.

How is LLM tracing different from APM tracing?

APM tracing focuses on service health metrics such as latency, throughput, errors, and infrastructure behavior. LLM tracing adds semantic AI context, including prompts, retrieved documents, tool inputs and outputs, completions, tokens, and scoring signals, which are needed to debug answer quality.

Which is the best LLM tracing tool?

Braintrust is the best LLM tracing tool for teams that want tracing connected to evaluation, online scoring, monitoring, and regression testing in a single workflow. Braintrust captures production traces as structured spans, supports SDK and OpenTelemetry paths, and lets teams turn real failures into eval coverage. For a detailed comparison, see Best LLM tracing tools.