> ## Documentation Index
> Fetch the complete documentation index at: https://www.braintrust.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Optimize costs

> Optimize your Braintrust usage and LLM spend with less duplicate logging, sampled online scoring, cheaper models and judges, and batch evals.

Your costs come from two sources:

* **Braintrust bill**: Charges based on your [usage](https://www.braintrust.dev/docs/plans-and-limits.md#usage-limits), such as the data you send and the scores you record.
* **LLM spend**: The cost of the model calls that your application, evaluations, and scorers make.

Each approach below reduces one or both. Evaluations show whether a lower-cost option still meets your quality bar.

| Approach | What it reduces | Best for |
| - | - | - |
| [Reduce duplicate logging](#reduce-duplicate-logging) | Braintrust bill (processed data) | Applications with overlapping instrumentation or OpenTelemetry tracing |
| [Control online scoring volume](#control-online-scoring-volume) | Braintrust bill (recorded scores) and LLM spend (judge calls) | High-volume production traffic |
| [Choose lower-cost models](#choose-lower-cost-models) | LLM spend (model calls in your application and evaluations) | Tasks where a smaller or open-weight model meets your quality bar |
| [Optimize scorer calls](#optimize-scorer-calls) | LLM spend (scorer calls in evaluations and online scoring) | Deterministic or clearly bounded checks, and judges that read whole traces |
| [Run evaluations through batch APIs](#run-evaluations-through-batch-apis) | LLM spend (task and scorer calls in evaluations) | Large evaluations that can wait for results |

## Optimize your Braintrust usage

Processed data is [measured at ingestion](https://www.braintrust.dev/docs/admin/billing/faq.md#does-deleting-data-reduce-my-usage), so deleting logs or shortening retention doesn't lower it. Your bill goes down when you send less duplicate data and record fewer scores.

### Reduce duplicate logging

Duplicate logging sends the same data more than once, and every copy counts toward processed data. It can come from overlapping instrumentation or from repeating a call's details on more than one span. Spans unrelated to your LLM calls add to processed data the same way.

> **Trace each call with one layer**
>
> When more than one tracing layer instruments the same LLM call and each sends spans to Braintrust, the call is logged once per layer. Keeping one layer per call avoids the duplicates.
>
> For example, [LiveKit Agents](https://www.braintrust.dev/docs/integrations/agent-frameworks/livekit-agents.md) can send traces to Braintrust through LiveKit's built-in OpenTelemetry instrumentation or through Braintrust's auto-instrumentation. If you use LiveKit's OpenTelemetry pipeline and also run auto-instrumentation for other libraries, turn off Braintrust's LiveKit integration so each session is traced once.
>
> How you turn off an integration depends on the SDK:
>
> * **TypeScript**: List the integrations to skip in the `BRAINTRUST_DISABLE_INSTRUMENTATION` environment variable. For LiveKit, this also disables `wrapLiveKit()`. See [Disabling specific integrations](https://www.braintrust.dev/docs/instrument/trace-llm-calls.md#disabling-specific-integrations).
>
>   ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
>   BRAINTRUST_DISABLE_INSTRUMENTATION=livekit node --import braintrust/hook.mjs app.js
>   ```
>
> * **Python**: Pass `False` for the integration's keyword to `auto_instrument()`. See [Disabling specific integrations](https://www.braintrust.dev/docs/instrument/trace-llm-calls.md#disabling-specific-integrations-1).
>
>   ```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
>   braintrust.auto_instrument(livekit_agents=False)
>   ```

> **Log each payload and metric on one span**
>
> Copying an LLM call's prompt, response, or token metrics onto its parent span sends the same data twice, and both copies count toward processed data. Preset cost charts also add up the token metrics on every span, so they show the call's cost twice, even though the provider billed it once.
>
> Log the token metrics and the model only on the span that made the LLM call. See [Option 1: Log model on token-reporting spans](https://www.braintrust.dev/docs/kb/total-llm-cost-preset-requirements#option-1-log-model-on-token-reporting-spans).

> **Send only AI spans from OpenTelemetry**
>
> General-purpose OpenTelemetry instrumentation also exports spans unrelated to LLM calls, such as HTTP requests and database queries. The Braintrust span processor can send only AI-related spans. See [Braintrust span processor](https://www.braintrust.dev/docs/integrations/sdk-integrations/opentelemetry/send-traces-and-logs.md#braintrust-span-processor).
>
> How you configure the filter depends on the SDK:
>
> * **TypeScript**: Set `filterAISpans` to `true`. For finer control, pass a `customFilter` function.
>
>   ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
>   new BraintrustSpanProcessor({
>     parent: "project_name:your-project-name", // Replace with your project name
>     filterAISpans: true,
>   });
>   ```
>
> * **Python**: Set `filter_ai_spans` to `True`. For finer control, pass a `custom_filter` function.
>
>   ```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
>   provider.add_span_processor(BraintrustSpanProcessor(filter_ai_spans=True))
>   ```

> **Preserve raw OTel attributes only for debugging**
>
> Braintrust removes OpenTelemetry attributes from `metadata` once it maps them to fields such as `input` and `output`, so the same payload isn't stored twice. This is automatic on Braintrust-hosted deployments, and self-hosted deployments need to opt in. See [Strip OTel attributes from metadata](https://www.braintrust.dev/docs/kb/strip-otel-attributes-from-metadata).
>
> Setting `braintrust.otel.preserve_attributes` to `true` on a span keeps the raw attributes in `metadata` alongside the parsed fields, so the span stores them twice. Set it only on spans you're debugging:
>
> * **TypeScript**: Call `setAttribute()` on the span.
>
>   ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
>   span.setAttribute("braintrust.otel.preserve_attributes", true);
>   ```
>
> * **Python**: Call `set_attribute()` on the span.
>
>   ```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
>   span.set_attribute("braintrust.otel.preserve_attributes", True)
>   ```

To confirm that a change removed duplicates, compare spans per trace before and after the change. See [Braintrust usage](#braintrust-usage).

### Control online scoring volume

Each score you record counts toward your plan's [score usage](https://www.braintrust.dev/docs/plans-and-limits.md#usage-limits). [Online scoring](https://www.braintrust.dev/docs/evaluate/score-online.md) rules run their scorers on each sampled production trace or span, so a rule's **Sampling rate** controls both the number of scores you record and the model spend of its LLM-as-a-judge scorers.

* **Lower the sampling rate for high-volume traffic.** High-volume applications can use rates of 1-10% to manage costs. Low-volume or critical applications can use 50-100% for comprehensive coverage.
* **Score only the traces and spans you need.** Use a rule's **Filter** setting to choose which traces or spans it scores, such as spans with specific names. See [Configuration parameters](https://www.braintrust.dev/docs/evaluate/score-online.md#configuration-parameters).
* **Match the scope to the judgment.** A span-scoped rule scores every matching span, so one trace can produce many scores. A trace-scoped rule scores each sampled trace once. New instrumentation can add spans that match an existing span-scoped rule and raise scorer volume without any change to the rule.
* **Pause rules you don't need.** A paused rule stops scoring new traces and keeps its scorers, filters, and scope. Traces logged during the pause aren't scored later. See [Pause an automation](https://www.braintrust.dev/docs/evaluate/score-online.md#pause-an-automation).

To change a rule's sampling rate, go to ** Settings** > [** Automations**](https://www.braintrust.dev/app/~/configuration/automations), open the rule under **Online scoring**, and update **Sampling rate**.

To lower the model cost of each judge call, see [Optimize scorer calls](#optimize-scorer-calls).

## Optimize LLM costs

Evaluations show whether a lower-cost model or scorer holds quality on your own data before you change production. The evaluations themselves can also run for less.

### Choose lower-cost models

A smaller model often handles a well-defined task as well as a frontier model, at a lower per-token price. Braintrust serves several open-weight models, including GLM, Kimi, and DeepSeek models, as [built-in models](https://www.braintrust.dev/docs/admin/ai-providers.md#available-models) that you can use without configuring your own AI provider. Their usage draws down your organization's model credits.

Per-token rates vary widely, and not every open-weight model costs less than a comparable proprietary model. [Model credits](https://www.braintrust.dev/docs/plans-and-limits.md#model-credits) lists the rate for each built-in model. Built-in models also have [plan and deployment requirements](https://www.braintrust.dev/docs/admin/ai-providers.md#requirements).

A [playground](https://www.braintrust.dev/docs/evaluate/playgrounds.md) compares models side by side on the same dataset and scorers. To compare a candidate model with your current one:

1. In a playground, add a prompt task for your current model and another for the candidate, with the same prompt, dataset, and scorers. Built-in models appear under the **Braintrust** provider.
2. Select  **Run**. Each output in the grid shows its estimated LLM cost.
3. Turn on [diff mode](https://www.braintrust.dev/docs/evaluate/playgrounds.md#compare-with-diff-mode) with your current model as the base task. Each candidate shows score changes, timing and token usage differences, and a comparison grade of **Improvement**, **Regression**, **Tradeoff**, or **Tie**.
4. To compare estimated cost across the whole dataset, select **+ Experiment**. Each task becomes its own experiment, so you can [compare the experiments](https://www.braintrust.dev/docs/evaluate/compare-experiments.md#assess-overall-impact) and review scores, estimated cost, and latency together.

### Optimize scorer calls

Scorers that call an LLM add a model call for every case they grade, in both experiments and online scoring.

> **Choose cost-effective scorers**
>
> Scorers that don't call a model, such as [custom code](https://www.braintrust.dev/docs/evaluate/custom-code.md) checks and heuristic [autoevals](https://www.braintrust.dev/docs/evaluate/autoevals.md) like `Levenshtein`, add no model cost and run faster. They suit checks with a deterministic answer, such as exact matches, valid JSON, required fields, or numeric tolerances. [LLM-as-a-judge](https://www.braintrust.dev/docs/evaluate/llm-as-a-judge.md) scorers are better for criteria that need judgment.
>
> When a check needs a judge, the judge model sets its cost. For frequent, focused checks with a clear answer, a structured decision model costs less per call than a general-purpose LLM. [TypeSafe Jev](https://www.braintrust.dev/docs/integrations/ai-providers/typesafe.md#llm-as-a-judge) selects one of the answers you define instead of generating text, and Braintrust turns the selection into a score or a classification label.
>
> Built-in Jev draws down your model credits at a low per-token rate, listed in [Model credits](https://www.braintrust.dev/docs/plans-and-limits.md#model-credits). It must be enabled for your organization and isn't available for BYOC or self-hosted deployments. See [Configure access](https://www.braintrust.dev/docs/integrations/ai-providers/typesafe.md#configure-access).
>
> For broad or nuanced criteria, such as tone or helpfulness, a general-purpose LLM is the better judge. Comparing candidate judges against human-assigned scores shows whether a lower-cost judge is accurate enough. See [Choose a judge model](https://www.braintrust.dev/docs/evaluate/llm-as-a-judge.md#choose-a-judge-model).

> **Reduce judge input**
>
> A [trace-level](https://www.braintrust.dev/docs/evaluate/llm-as-a-judge.md#score-traces) LLM-as-a-judge scorer reads the trace through template variables such as `{{thread}}`, and everything in them adds input tokens to each judge call. The scorer's [preprocessor](https://www.braintrust.dev/docs/observe/topics/enable.md#write-a-custom-preprocessor) builds these variables, so a custom preprocessor can skip spans the judge doesn't need, such as middleware, or shorten large tool call payloads.
>
> To set a scorer's preprocessor, select one under **Preprocessor** when you edit the scorer, or use [`bt scorers create --preprocessor`](https://www.braintrust.dev/docs/reference/cli/scorers#preprocessors). Scorers without their own preprocessor use the project's [default preprocessor](https://www.braintrust.dev/docs/admin/projects.md#set-a-default-preprocessor), which Topics facets also use.

### Run evaluations through batch APIs

> [!WARNING]
>
> Workflow evaluations are in [public preview](https://www.braintrust.dev/docs/feature-lifecycle.md) and can change before reaching general availability.

Model providers such as OpenAI and Anthropic offer batch APIs that process requests asynchronously, often at a discount. Workflow evaluations send an evaluation's task or scorer requests through these APIs, collect the results by polling or through a webhook, and record the outputs and scores in one Braintrust experiment.

Batch processing suits large regression suites, model comparisons, and scoring jobs that don't need immediate results. Results arrive when the provider completes the batch, which can take hours.

Workflow evaluations require Python SDK v0.39.0 or later, TypeScript SDK v3.37.0 or later, or Go SDK v0.18.0 or later. The [Python](https://www.braintrust.dev/docs/sdks/python/workflow-evaluations.md), [TypeScript](https://www.braintrust.dev/docs/sdks/typescript/workflow-evaluations.md), and [Go](https://www.braintrust.dev/docs/sdks/go/workflow-evaluations.md) guides walk through a complete OpenAI Batch API example.

## Measure your savings

Usage and cost rise and fall with traffic. Comparing per-trace numbers across equivalent periods, such as the same days of the week, separates real savings from changes in volume. See [Calculate per-trace averages in dashboards](https://www.braintrust.dev/docs/kb/compute-average-cost-and-llm-calls-per-trace).

### Braintrust usage

**Usage analytics** on the ** Settings** > [** Billing**](https://www.braintrust.dev/app/~/configuration/billing) page charts daily logs in GB, scores, and Topics tokens for your whole organization or by project. See [Monitor usage](https://www.braintrust.dev/docs/admin/billing/monitor-usage.md#view-usage).

Unlike total usage, spans per trace doesn't rise with traffic, so an increase points to a change in your application or its tracing, such as two layers tracing the same calls. To track it, use the [average spans per trace](https://www.braintrust.dev/docs/reference/sql/best-practices.md#per-trace-averages) query.

### LLM spend

Braintrust estimates the cost of each LLM call from its token counts and the model's pricing, so you can see the effect of each change:

* **Experiments**: The **Estimated cost** column on the [** Experiments**](https://www.braintrust.dev/app/~/experiments) page and the comparison summary layouts show cost next to scores. See [Assess overall impact](https://www.braintrust.dev/docs/evaluate/compare-experiments.md#assess-overall-impact).
* **Dashboards**: The built-in **Cost and quality** dashboard tracks cost and scores over time across your logs and experiments. See [Dashboards](https://www.braintrust.dev/docs/observe/dashboards.md).
* **Scheduled reports**: The **Weekly cost report** [Loop automation recipe](https://www.braintrust.dev/docs/loop/automations.md#loop-automation-recipes) posts estimated cost by user to Slack each week, compared with the previous week.
* **Traces**: Each span shows its estimated cost, and parent spans include the cost of their children. See [Examine traces](https://www.braintrust.dev/docs/observe/examine-traces.md).

If the same call's token metrics are logged on more than one span, preset cost charts show its cost more than once, even though the provider billed the call once. See [Reduce duplicate logging](#reduce-duplicate-logging).

> [!NOTE]
>
> The **Estimated cost** column on the Experiments page and the preset cost charts on dashboards exclude scorer spans. To see what your scorers cost, [query scorer spans with SQL](https://www.braintrust.dev/docs/kb/query-scorer-span-token-costs-with-sql).

## Next steps

* [Compare experiments](https://www.braintrust.dev/docs/evaluate/compare-experiments.md) to weigh cost against quality.
* Run your first batch evaluation with the [Python](https://www.braintrust.dev/docs/sdks/python/workflow-evaluations.md), [TypeScript](https://www.braintrust.dev/docs/sdks/typescript/workflow-evaluations.md), or [Go](https://www.braintrust.dev/docs/sdks/go/workflow-evaluations.md) workflow evaluations guide.
* [Monitor your usage](https://www.braintrust.dev/docs/admin/billing/monitor-usage.md) and set up spend alerts.
* Set up [online scoring](https://www.braintrust.dev/docs/evaluate/score-online.md) rules for production traffic.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.