Discover

Find behavior that needs attention

Automatically surface important agent behavior, investigate the evidence, and turn what you learn into measurable improvements.

Start discovering Contact sales

Understand what happened and what to improve

When your agents handle thousands of requests each day, you can’t read every trace. Braintrust continuously applies intelligence to production data to surface important behavior, investigate the evidence, and help you decide what to improve.

Tools for understanding agent behavior

From production traces to measurable improvements

Find runs where the agent completed the requested work but failed to return it, create a regression dataset and scorer from those examples, and configure monitoring for the behavior.

I'll use Braintrust's pattern-analysis workflow to identify high-confidence examples, then the evaluator workflow to build and validate a regression scorer, and finally the automations workflow to configure ongoing monitoring. I'll use the current project already selected in Loop.

Used 3 tools6s

I'm starting with read-only SQL triage and raw trace inspection. I'll look for the specific signature “work completed in tools/state, but the root response is empty or never communicates the result,” and I'll include contrasting successful handoffs so the scorer does not merely flag every tool-heavy run.

Used 7 tools33s

Turn ID: e900c7e4-dc4a-5f2f-99d1-dd3cf14a362c

The existing Pattern is an exact match, so I'll update it rather than create a duplicate. The recent blank roots are credential failures with no completed tool work, so I'm excluding those; I'm now inspecting the Pattern's two true positive traces and comparable successful handoffs to ground the dataset and rubric.

Used 3 tools13s

The grounded rubric is trace-scoped: score 0 when the agent begins or completes work but the trace ends without a user-facing result; score 1 when the result is clearly handed back; skip requests that require no work/result handoff. I'm testing it against the two failures and two successful order workflows before saving anything.

Tested evaluator

Arguments

{
  "approval_justification": "Retry Final Handoff calibration on one known missing-handoff failure after the previous batch timed out.",
  "function": {
    "inline_evaluator": {
      "name": "Final Handoff",
      "allow_skip": false
    }
  },
  "project_id": "8dd551aa-6da8-4602-b7c5-fc72a6121c00",
  "test_case": {
    "label": "Known missing-handoff failure",
    "expected_score": 0
  }
}

Output

{
  "operation": "test_evaluator",
  "project_id": "8dd551aa-6da8-4602-b7c5-fc72a6121c00",
  "results": [
    {
      "expected_score": 0,
      "score": 0,
      "passed": true
    }
  ],
  "summary": {
    "passed": 1,
    "failed": 0
  }
}

Tested evaluator

Arguments

{
  "skip_realtime": true
  }
},
],
"project_id": "8dd551aa-6da8-4602-b7c5-fc72a6121c00"
}

Output

{
  "operation": "test_evaluator",
  "project_id": "8dd551aa-6da8-4602-b7c5-fc72a6121c00",
  "results": [
    {
      "expected_score": 0,
      "score": 0,
      "passed": true
    }
  ],
  "summary": {
    "passed": 1,
    "failed": 0
  }
}

Final status: I calibrated Final Handoff on four representative traces: two missing-handoff failures scored 0, and two successful handoffs scored 1. I saved the regression dataset and evaluator, then added monitoring for this configuration.

Ask questions about logs, @ for context, / for commands

Loop

Ask your data anything

Ask open-ended questions across traces, logs, experiments, and datasets. Loop can run SQL and code, verify findings against additional evidence, repeat investigations on a schedule, and take approved action.

Explore your data

f2ffd8bb·2hr

Return-policy answers invent the window start date

Summary

Return-policy responses still sometimes add a start date that is not present in the `checkReturnPolicy` tool output. In the selected 30-day window, at least one conversation says the 30-day electronics return window starts from purchase, while the tool only returns `returnWindow: "30 days"`. This can misstate policy eligibility when purchase, shipment, and delivery dates differ.

Suggested fix

Update the support prompt or response formatter so the agent repeats the tool's returnWindow literally unless the tool returns an explicit windowStartsAt field. Better yet, extend checkReturnPolicy to return the window anchor as structured data and add an evaluator for unsupported policy qualifiers.

Monitors

Conversation Outputs With Raw Suggestion Markup

43210

Aug 1Aug 2Aug 3Aug 4Aug 5Aug 6Aug 7

Traces

conversation32b59c9b

The tool returns only a 30-day window, while the assistant adds a date-of-purchase anchor.

Evidence

Tool policy window

The return-policy tool did not include a start anchor.

30 days

Unsupported anchor in final answer

The final answer adds a policy anchor that is absent from the tool output.

30 days from the date of purchase

Patterns

Find recurring behavior worth investigating

Patterns is a Loop automation that identifies recurring issues and opportunities across production traces. Each Pattern includes supporting evidence, its impact, and a recommended next step.

Review your patterns

Task

User intent or goal

24.0%203 items

Refund requests

Customers are asking for money back after missing deliveries, delayed refund status updates, and policy confusion around damaged or returned orders.

18.4%156 items

Sizing and fit questions

Shoppers need help choosing sizes, comparing fit across products, and deciding whether an item will work before placing or exchanging an order.

15.0%127 items

Order tracking

Users want current shipment status, tracking links, delivery estimates, and explanations for packages that appear stalled or delayed in transit.

10.9%92 items

Damaged items

Customers report products arriving broken, packaging failures, and requests for replacements where the agent needs to gather evidence and route next steps.

9.0%76 items

Subscription questions

Users ask how renewals, cancellations, and subscription refunds work, often needing policy-grounded answers before support escalation.

Topics

Organize broad themes across traces

Topics classifies production traces across tasks, issues, sentiment, and custom facets. Use it to understand common behavior across large volumes of traffic and query the results with SQL.

Classify your logs

Debugger analysisRun debugger

Summary

This trace contains evidence for 1 reported pattern.

Facet span metrics

Medium

unknown

Hypothesis

A high-token Facets span analyzed an access-request/helpdesk artifact rather than an Acme support conversation.

Evidence

Facet span metricsView span

49355

Facet outputView span

test application

Facet automation do...

Debugger

Explain complex behavior in individual traces

Debugger analyzes spans, tool calls, tool results, and model outputs to identify likely failure modes. It grounds each explanation in trace evidence and recommends what to investigate next.

Debug your traces

Built for agent data at scale

Discovery only works if queries stay fast at production scale. Brainstore is Braintrust's database for agent observability. Search and filter millions of traces in under a second, including full-text search across prompts and error messages.

0.0x

Faster full-text search

Competition

0 ms

Brainstore

0 ms

0.00x

Faster write latency

Competition

0 ms

Brainstore

0 ms

0.00x

Faster span load time

Competition

0 ms

Brainstore

0 ms

Learn more about Brainstore

Continue the improvement loop

Measure what happens next

Turn production evidence into classifiers, scorers, eval datasets, dashboards, monitors, and alerts. Track whether the next version improves quality without increasing latency or cost.

Alerts

Notifications when scores, latency, or errors cross boundaries.

Set up alerts

Online scoring

Production traffic scored continuously. Same scorer library.

Score production traffic

Quality gates

Block deployments when eval scores drop below thresholds.

Add to CI

Sandboxed evals

Push eval code once. Teammates run complex agents from the playground without local setup.

Run sandbox evals

Customer stories

From reactive to proactive

Meet all the teams

Sarah Sachs, AI Lead

“There are some problems we wouldn't know were problems without Braintrust.”

Play

Luis Héctor Chávez, CTO

“Braintrust helped us identify several patterns that we wouldn't have found.”

Play

Allen Kleiner, AI Engineering Lead

“Loop helps us understand trace details that would be impossible to scan manually.”

Play

Start building

Free to start. No credit card required.

Start discovering Contact sales