What is LLM cost management? Tracking tokens, usage, and spend

How to investigate an LLM bill increase

If last month's bill has already risen, start with provider usage and billing data for the affected period. Tracing tools can explain only the application-level details that were recorded at the time.

1. Export daily usage by model, API key or project where the provider exposes it, and token category—input, output, cached input, and reasoning where reported. Reconcile the totals to the invoice; available breakdowns vary by provider.
2. Find when spend or cost per request increased. Separate higher request volume from higher cost per call.
3. Compare that date with deployments, prompt or retrieval changes, model changes, traffic and tenant mix, retries, and agent steps.
4. If cached-input usage fell while uncached input rose, compare the same model and a similar workload, then check cache writes and prompt-prefix changes before concluding that caching broke.

Provider exports are useful for retrospective, provider-level triage, but they may not identify the application feature or tenant behind each call. Tags cannot reconstruct that attribution for requests that were never tagged. Once call-level tracing and metadata are in place, Braintrust can retain those dimensions on future traces for investigation.

What is LLM cost management?

LLM cost management is the process of planning and controlling model inference spending throughout an application's lifecycle. It helps engineering, product, and finance determine whether an AI feature can operate within its budget and meet its quality requirements.

Measurement: Establish the cost of model calls using recorded token usage and applicable rates. Engineering maintains the instrumentation and pricing data needed for reliable estimates.

Attribution: Assign spending to the users, customer accounts, features, prompt versions, and agent runs responsible for it. Request-level attribution supports product pricing and analysis of account profitability.

Forecasting: Project future spending from observed costs and expected usage, accounting for planned launches or model changes. Finance uses these projections to prepare budgets and estimate margins.

Control: Define spending thresholds and enforce limits on requests, tokens, or agent iterations in the application or gateway. Alerts notify the responsible people when usage needs investigation.

Optimization: Identify changes that reduce the cost of delivering an acceptable result. Evaluations establish whether proposed savings preserve the application's required behavior before rollout.

LLM cost management vs. LLM cost tracking and LLM cost monitoring

The three terms describe different scopes, and teams often use them interchangeably when deciding what to build.

LLM cost tracking is the measurement layer: capturing token usage per call, applying provider rates, and recording the result alongside the request so spending can be queried later. Tracking answers what a call cost and which feature, tenant, or prompt version produced it.

LLM cost monitoring adds the ongoing observation layer: charts over time, thresholds, baselines, and alerts that fire when spending or cost per request moves. Monitoring answers whether current spending is normal and who should look at it.

LLM cost management is the discipline that uses both, plus the decisions around them: setting budgets against named workloads, forecasting the next period, enforcing caps, and evaluating whether a cheaper configuration still meets the quality requirement before it ships. Tracking and monitoring produce the evidence; management is what the team does with it.

Also read: How to track LLM costs and What is LLM monitoring?

Why provider invoices miss LLM cost drivers

Provider invoices summarize billed usage and charges over a billing period. Explaining those charges requires application context that billing records often lack, especially when multiple customers and features share the same provider account or API credentials.

Customer and tenant: Shared billing can combine usage from many customer accounts. Without customer identifiers attached to requests, the total cannot establish how much each account costs to serve or whether its contract covers that expense.

Feature and prompt version: Higher spending may follow a feature launch, a longer system prompt, or a retrieval change that adds context to every call. Request records tied to feature and prompt versions help distinguish increased traffic from a more expensive configuration.

Agent behavior: One user request can trigger retries, repeated tool calls, and sub-agent activity. Aggregate charges conceal the execution sequence needed to identify unnecessary calls or investigate why an agent kept running.

Output value: Two configurations with equal spending can deliver different numbers of usable results. Connecting cost to task outcomes and evaluation scores reveals how much the application spends to produce an acceptable result, including the model calls consumed by unsuccessful attempts.

How LLM costs are calculated per request

Calculate each model call's cost by multiplying usage in each billing category by its applicable rate, then adding the charges. For prices quoted per million tokens, divide token counts by one million before applying the rate. A user request's total includes every billable call it triggers, plus any separate tool or service fees.

Input and output tokens: Input includes system instructions, conversation history, retrieved context, and tool definitions. Output includes generated tokens, which often cost more. Longer answers can therefore add more to a request's cost than the same number of input tokens would.

Model tier: Each model has its own rates. Anthropic lists base input prices of $0.10 per million tokens for Claude Haiku 5.5 prompts up to 100,000 tokens and $10 for Claude Fable 5.1, a hundredfold difference before other pricing adjustments.

Cached input: Reused prompt prefixes can receive discounted read rates, with writes sometimes charged at a premium. Anthropic's prompt caching rates charge 1.25 times base input pricing for five-minute cache writes and twice base pricing for one-hour writes. Reads cost one-tenth of base input pricing or less, depending on the model. OpenAI's cache charges also vary by model generation. Separate cached portions from any input total that already includes them before pricing the remaining input.

Reasoning tokens: Internal reasoning can incur charges even when the visible answer is short. OpenAI bills reasoning tokens as output, and Gemini pricing includes thinking tokens in output charges. When reasoning usage is already included in a billed output total, adding it again would double-count the cost.

Retries: Additional attempts increase spending when they incur billable usage. An answer rejected by the application can still carry charges, so request totals need to include unsuccessful attempts as well as the final accepted response.

Tool loops and agent steps: Agent workflows can trigger repeated model calls with growing conversation history and tool results. Each billable call contributes to the run's total, even when earlier context receives a cache discount. Provider-hosted tools may also have separate usage fees.

Provider differences: Tokenizers, service tiers, batch discounts, and context-length thresholds affect the calculation. Some Gemini models charge higher rates above 200,000 prompt tokens, and Claude Haiku 5.5 applies higher rates above 100,000. Use the rates and usage conventions for the specific model and endpoint.

Also read: How to track LLM token usage

LLM unit cost metrics beyond total spend

Unit cost metrics show how much an application spends to serve a particular account or complete a defined unit of work, making comparisons useful as traffic changes. Keep the reporting period and workload consistent so each metric includes the relevant model calls and outcomes.

Typical and unusually high costs: Report the median alongside p95 or p99 across requests, runs, or accounts, as appropriate. The median shows typical spending, while higher percentiles reveal expensive cases that an average can obscure. At p95, 95% of observations cost that amount or less.

Cost per quality-approved output: Divide total model spend for the evaluated workload by the number of outputs that pass the quality threshold. Include spending on failed attempts and retries in the numerator. A model with a lower per-call price can still cost more per approved output when its failure rate is higher.

Core components of LLM cost tracking and monitoring

A cost tracking and monitoring system should connect spending summaries to the requests behind them, giving the person investigating an alert a direct route to the model calls and application behavior responsible for the increase.

Token, cost, and latency capture

Capture provider-reported usage for each model call, preserving input, output, cache, and reasoning details where available. Recording the model, estimated cost, latency, and time to first token on the same span keeps spending and response speed comparable during an investigation.

Low-overhead logging: Buffer records and export them in asynchronous batches so network activity adds less delay to application responses, with export failures handled separately from application errors. Flush pending records during shutdown, and ensure streaming integrations collect the provider's final usage data before completing the log.

Tagging and cost attribution

Attach user, tenant, feature, environment, prompt version, and agent identifiers when a request enters the application, then propagate that context through tool calls and sub-agent work so costs remain attributable across the full execution.

For a custom client, create an LLM span around each call and log the feature and tenant on that same span alongside provider usage. The Braintrust Python SDK documents this start_span() pattern; use the provider's supported wrapper when available, and avoid logging the same model call twice.

from braintrust import start_span


def answer(request, tenant_id):
    with start_span(type="llm", name="support-answer") as span:
        response = my_llm_client.chat(request)
        # The custom client adapter must normalize provider usage
        # into Braintrust's metric names and token-total conventions.
        usage = response["usage"]
        metrics = {
            "prompt_tokens": usage["prompt_tokens"],
            "completion_tokens": usage["completion_tokens"],
            "tokens": usage["total_tokens"],
        }
        # Preserve reported breakdowns; missing usage is not zero.
        for field in (
            "prompt_cached_tokens",
            "prompt_cache_creation_tokens",
            "prompt_cache_creation_5m_tokens",
            "prompt_cache_creation_1h_tokens",
            "completion_reasoning_tokens",
        ):
            if usage.get(field) is not None:
                metrics[field] = usage[field]

        span.log(
            input=request["input"],
            output=response["output"],
            metadata={
                "model": request["model"],
                "feature": "support-answer",
                "tenant": tenant_id,
            },
            metrics=metrics,
        )
    return response

This example assumes the custom client adapter maps provider-specific usage fields to Braintrust's normalized metrics. prompt_cached_tokens records cache reads, and prompt_cache_creation_tokens records cache writes. Normalize prompt_tokens to include uncached input, cache reads, and cache writes; do not add the cache breakdowns again to that total. When the provider reports separate five-minute and one-hour cache writes, supply prompt_cache_creation_5m_tokens and prompt_cache_creation_1h_tokens instead of the aggregate write field, preserving their different pricing tiers—the LangChain integration and the Anthropic integration both break cache creation out this way. Log reasoning usage as completion_reasoning_tokens, which Braintrust treats as a subset of completion_tokens rather than an additional charge. Omit unreported breakdowns rather than substituting zero, and use a supported provider wrapper where possible to handle normalization.

In a dashboard, group estimated cost and request count by metadata.feature, compare cost per request, and open the contributing traces. This attribution starts with instrumented calls; it does not backfill older requests.

Cardinality: Use fields with relatively few distinct values, such as feature, model, and environment, to group dashboard results and define alert dimensions. User and agent run IDs are more useful for narrowing an investigation to specific accounts or executions, since charting every distinct value can crowd dashboards with thousands of series.

Normalized pricing across providers

Maintain a pricing registry that covers each model's billing categories, service tiers, and context-length thresholds, converting usage into a consistent currency and pricing units before comparing providers.

Price table ownership: Assign an owner to review provider changes and update the registry, retaining enough pricing context to explain how historical estimates were calculated.

Discounts and custom models: Represent batch discounts, negotiated rates, and custom model pricing explicitly, recording the calculated cost with each call when token rates cannot capture the actual charge. Reconcile estimates with billed totals to identify missing usage or differences between recorded and contracted pricing.

DIY logging table or cost-management platform?

A DIY setup can be sufficient when the team can maintain its instrumentation, pricing data, and investigation workflow. A platform may reduce that maintenance, but verify what it captures and how its estimates are calculated.

Neither approach makes estimates automatically exact: reconcile tracked costs with provider bills.

Cost dashboards

Show spending over time with breakdowns by model, feature, tenant, environment, and prompt version. Include request volume, unit costs, quality scores, latency, and errors for the same workload so reviewers can distinguish traffic growth from changes in the cost or behavior of individual requests.

Drill-down: Connect chart results to their underlying traces so an investigator reviewing rising feature costs can inspect the requests, model calls, and prompt versions contributing to the increase.

Budgets and spend forecasting

Estimate period spending from expected request volume and observed average cost per request, segmented by feature or workload and adjusted for planned launches, model changes, and anticipated usage growth.

Soft and hard limits: Define notification thresholds at specified portions of the budget, and enforce blocking or throttling in the application or gateway before additional model calls begin. Set per-user caps to bound trial usage and align paid-account limits with contract terms.

Agent iteration caps: Check step, tool-call, and token budgets in the agent's execution logic before additional steps begin, so limits contain spending during a run. Use alerts to support investigation after usage has been recorded.

Anomaly detection and cost alerts

Compare unit costs with an appropriate baseline, such as the same feature and model under normal traffic, to detect changes that remain below a fixed budget threshold. A gradual rise in cost per request may reveal a prompt regression or declining cache reuse before total spending triggers a budget alert.

Per-request alerts flag unusually expensive executions, and windowed alerts assess aggregates such as hourly feature spending or daily budget consumption. Braintrust separates these as log alerts and time window alerts: a log alert evaluates a SQL filter against individual production logs, so estimated_cost() > 1.0 catches a single expensive request, while a time window alert fires when a SQL calculation over a window crosses a threshold. Reviewing flagged executions and windows alongside latency and error signals points to causes such as repeated failed attempts.

Route notifications to the responsible team through Slack, PagerDuty, or a webhook, including the affected feature, model, environment, observed value, and threshold or baseline. Links to representative traces let the responder begin investigating directly from the notification.

Also read: Best tools for tracking LLM costs in production

LLM cost vs. quality, latency, and reliability

Compare current and candidate configurations on the same inputs, using consistent quality criteria and timing measurements. Spending can fall because a change improves efficiency or because the application produces shorter, incomplete, or less reliable results, so cost alone cannot establish whether the change benefits users.

Quality requirements: Check release-critical categories separately, including uncommon cases with serious consequences. An overall pass rate can remain stable even as performance deteriorates for a particular task or customer group.

Latency requirements: Additional reasoning or context may improve quality at the expense of cost and response time. Batch processing can reduce rates by accepting asynchronous completion, making it useful for work that can tolerate a delay. Interactive requests still need to meet the product's response-time requirements.

Reliability requirements: Include completion and retry rates when assessing savings. More attempts to produce an acceptable result can erase a lower per-call price and increase user wait times.

LLM cost optimization strategies

Choose a strategy based on what expensive traces reveal, such as unnecessary context, repeated generations, or excessive agent steps, so the change addresses a measured source of spending.

Prompt and context reduction

Remove redundant instructions, outdated few-shot examples, and unnecessary tool descriptions, preserving the information needed to complete the task. For retrieval workflows, test smaller or fewer chunks, and consider summarizing older conversation turns when the full history is no longer useful.

Output limits work best when the task has a predictable response size, such as a classification label or a fixed set of structured fields. Compare reduced prompts and generation limits against the existing configuration using the same evaluation cases.

Prompt caching and response caching

For prompt caching, place stable instructions, tool definitions, and reference material in a reusable prefix that remains consistent across requests. Measure cache-read usage and actual cost together, accounting for write charges and prefixes that receive too little reuse to generate savings.

Response caching stores complete outputs and can eliminate another model call when an eligible request repeats. Use it for requests whose answers remain valid during the cache lifetime, with matching rules that account for the context affecting the answer. Bypass stored responses when an evaluation needs fresh generations or measures output variability.

Model routing for cost optimization

Model routing assigns requests to models that have demonstrated acceptable performance for the relevant task, and a routing policy can rely on several mechanisms.

• Rule-based routing: Assign defined request types to approved models, such as using a smaller model for classification and a stronger model for complex reasoning.
• Cascades: Start with a smaller model and escalate when a validator or quality check rejects its result.
• Classifier routing: Predict a request's category or difficulty before choosing an approved model for that class.

Evaluate each route on representative examples from its query class, including the routing decision and any escalation calls in the cost comparison. A cascade's total includes the initial attempt even when a larger model ultimately produces the accepted answer.

Batch processing

Batch APIs from OpenAI, Anthropic, and Google offer discounts for supported requests. Use asynchronous processing for offline evaluations, bulk classification, document backfills, and nightly enrichment, which allows the provider to complete requests within its completion window.

OpenAI specifies a 24-hour window, with unfinished requests expiring at the deadline. Track completed and failed items so recovery work is included in the job's total cost and turnaround time.

Retry and agent loop control

Retry transient failures, such as rate limits or temporary server errors, with backoff and bounded attempts. When an output fails validation, inspect whether another identical attempt can reasonably succeed or whether the prompt, input, or recovery procedure needs to change.

For agents, combine step, tool-call, and token budgets with detection of repeated identical actions. Use the distribution of successful runs, including unusually long executions, to choose initial limits, then test whether the caps interrupt work that would otherwise finish successfully.

Architecture changes

Move deterministic operations, including formatting, schema validation, date calculations, and rule-based extraction, into code. A workflow may also benefit from separating classification from generation or narrowing retrieval before the generation step, provided the additional calls do not outweigh the savings.

For sustained, high-volume tasks, evaluate whether a fine-tuned or distilled smaller model can meet the requirements currently served by a larger model. Compare the complete workflow because savings in one step can be offset by additional processing elsewhere.

The LLM cost management lifecycle

Cost work becomes repeatable when investigation, testing, gating, and monitoring use the same traces and the same quality criteria. The four stages below describe how that cycle runs in Braintrust.

Investigate spending with dashboards and Loop

Start from the built-in Cost and quality dashboard, which covers all data in a project and shows spending beside the quality scores for the same workload. The built-in dashboard is read-only; clone it or create your own to add charts broken down by metadata.feature, model, or tenant. Creating dashboards and custom charts is a Pro and Enterprise feature.

Loop, the Braintrust AI agent, answers cost questions in natural language against the same logs—breaking down cost by model over a time range, returning total, average, p50, p95, and p99 cost, or building a daily cost dashboard from a description. Loop can also hand a query off to the Query page so you can keep refining the SQL yourself. From either a chart or a Loop result, open the contributing traces to see the prompts, tool calls, and retries behind an expensive request.

Test cheaper configurations before rollout

Once an expensive pattern is identified, build the candidate change—a shorter prompt, a smaller model, a cache-friendly prefix—and run it against the same evaluation cases as the current configuration. Comparing experiments side by side shows the quality, cost, and latency differences together, so a saving that degrades a release-critical category is visible before it ships rather than after.

Gate changes in CI

The Braintrust eval GitHub Action runs evaluation suites on pull requests and posts a results comment that includes Cost and Duration alongside scores. By default, bt eval returns a non-zero exit code when an eval throws, which fails the job; defining a custom Reporter() lets the team decide precisely what counts as a failure—for example, a cost regression beyond an accepted margin. For CI systems other than GitHub Actions, invoke bt eval directly.

Monitor after rollout

Keep the dashboard and alerts pointed at the same dimensions used during the investigation, so a later regression surfaces against a known baseline. A log alert on estimated_cost() catches individual expensive requests as they happen, and a time window alert catches aggregate drift—daily spend rising above last week's average, for instance—before it reaches the budget threshold. Route both to the team that owns the feature, and include trace links so the responder starts from the evidence rather than from the chart.

Getting started with LLM cost management

LLM cost management starts with instrumentation, not with a spreadsheet. Capture token usage and estimated cost on every model call, attach the feature, tenant, and prompt version that produced it, and keep those records beside the quality scores for the same workload. From there, the discipline is ordinary engineering: investigate what the expensive traces actually show, test the cheaper configuration against the same evaluation cases, gate the change in CI, and monitor the dimension you changed.

Braintrust ties all four stages to the same traces. Start for free with 1 GB of processed data, or read how AI observability helps lower LLM cost at scale for a deeper look at the observability side.