Articles

What is prompt caching? How it reduces LLM cost and latency

13 November 2026Braintrust Team24 min
TL;DR: Prompt caching

Prompt caching reuses the computation a model has already performed for an identical prompt prefix. Reusing system instructions, tool definitions, examples, or earlier conversation turns can reduce input-token charges and time to first token. The model still processes new input and generates a fresh response, with output tokens billed at the applicable output rate.

This guide explains how to structure reusable prefixes, diagnose cache misses, and compare provider controls and pricing before measuring the cost and latency effects of caching. It also covers break-even calculations and provider-specific usage fields, explaining how to account for cache-write charges and avoid counting cached input twice when assessing savings.

Braintrust connects provider-reported cache usage with cost and latency on instrumented LLM calls. Traces and prompt-version metadata reveal which configurations reuse their prefixes and which changes increase input costs, so caching decisions can be checked against production results.


What is prompt caching?

Prompt caching is a technique in which an LLM provider retains intermediate computation for the beginning of a prompt and reuses it when a later request starts with the same tokens. The reusable portion is called the prompt prefix. It can contain system instructions, tool definitions, few-shot examples, reference documents, and earlier conversation turns. Google's Gemini API calls this feature context caching.

Cache hit: The provider identifies an eligible matching prefix and applies cached-input pricing to the reused tokens. Reusing the stored computation can reduce input-processing work, while the model continues with the new suffix and generates its response.

Cache miss: The provider processes the unmatched input. Eligible prefixes may then be written for later reuse, depending on the model's caching mode, breakpoints, and minimum length. Providers with a write premium bill those newly cached tokens at the cache-write rate.

The financial benefit depends on the share of input billed as cache reads and the cost of creating entries that are later reused. The latency benefit depends on how much input computation the inference service actually avoids; cached-token billing alone does not establish that a request ran faster.

What prompt caching reuses on a cache hit

Prompt caching applies to input processing. Understanding which computation is reused makes it easier to interpret both the bill and the latency measurements.

Cached KV states for the prompt prefix

As a transformer reads input tokens, its attention layers compute key and value vectors. Later tokens use the stored vectors to attend to earlier context during input processing and output generation. KV caching avoids recomputing those earlier vectors during generation.

Prompt caching extends reuse across requests by retaining the states associated with a shared prefix. A provider can resume input processing from an eligible cached prefix, then process the suffix added by the new request.

For a causal language model, a token's cached state depends on its preceding context. Changing an earlier token can therefore invalidate reuse after that position, even when the rest of the prompt looks unchanged. The reusable portion also depends on where the provider can find an existing cache entry.

Computation that still runs on every request

New input tokens: The model processes the suffix after the reused prefix, including the current user message, fresh retrieval results, and new tool output. Tokens in the suffix bill as ordinary input or as cache writes when selected for a new entry.

Output generation: The model generates a response using the complete request context. Prompt caching does not reuse the previous answer, and the provider bills generated tokens at the applicable output rate.

A response cache stores a complete answer and can return it without another generation call. Prompt caching keeps the generation step in place, but answer accuracy and freshness still depend on the instructions and source information supplied to the model.

Prompt prefix matching for system prompts, tools, and conversation history

Prompt caches require matching prefixes, so prompt order and the boundaries available for lookup determine how much input a later request can reuse.

Exact prefix matching and prompt order

Providers compare the rendered prefix, including content such as tools, instructions, text, and supported attachments. Groq's prefix matching requires an exact match, and Anthropic requires cached segments to be identical through the selected cache boundary.

Anthropic orders the context as tools, then system instructions, then messages. A changed tool definition can invalidate the otherwise unchanged instructions and messages after it. Request settings rendered into the context can also affect the match, even when the visible system-prompt text stays the same.

Matching content must also end at a boundary the provider can reuse. Anthropic writes at cache breakpoints and searches backward through up to 20 block positions, counting the breakpoint itself as the first. On OpenAI's GPT-5.6 and later models, lookup considers explicit breakpoints and, in implicit mode, additional eligible message endings. A long shared prefix can still miss when no reusable entry exists at the required boundary.

Prefixes that repeat across requests

System instructions and few-shot examples: Instructions, policies, and labeled examples that remain stable across requests can form a reusable prefix. Include examples that improve the task's results, then measure whether their reuse offsets the extra input volume.

Tool definitions: Names, descriptions, and schemas often repeat across agent steps. Keeping their content, order, and serialization stable keeps the prefix intact even as tool results and user messages change.

Conversation history: A multi-turn application can preserve earlier messages and append each new turn. The reusable prefix grows as the conversation develops, as long as previous context stays unchanged and eligible boundaries remain available.

Large static context: Several questions about one contract, manual, research paper, or codebase summary can reuse the same reference material. The follow-up question and any uncached suffix still incur input charges, and every generated answer incurs output charges.

Static-first prompt structure

Place stable content before content that changes on each request, within the ordering supported by the API. Reusable instructions, examples, schemas, and reference material belong ahead of the current question, timestamps, session variables, and unique request identifiers.

A changing value near the beginning can break the match for everything after it. On APIs with explicit breakpoints, mark the end of the stable prefix so unrelated questions can reuse the same instructions without requiring a match through the changing suffix.

Two requests sharing a cached prefix of tool definitions, system instructions, and few-shot examples, with the dynamic user message after the cache breakpoint

Common causes of prompt cache misses

A request can succeed and return a normal response even when its cached-token count is zero, so diagnose misses through usage fields and request configuration.

Dynamic content at the start: Timestamps, request IDs, or session variables inserted before shared instructions change the prefix on every request. Move changing values after the reusable content when the API and task permit.

No entry at the stable boundary: A breakpoint placed only after a changing suffix may write an entry that unrelated requests never match. Add a breakpoint at the end of the shared content. Growing conversations can reuse earlier writes, but Anthropic's 20-block lookback can miss an entry when too many blocks have been added since the previous breakpoint. On the Claude API, consecutive tool_use blocks count as one position and consecutive tool_result blocks count as one position, so parallel tool calls do not push earlier entries out of the window.

Changed tools or serialization: Editing a tool's name, description, parameters, or position alters the rendered prefix. Use deterministic serialization so equivalent tool schemas do not change representation between requests.

Changed request settings: Tool choice, image content, structured-output configuration, and model-specific thinking or effort settings can affect cached context. Compare the full request configuration when a stable-looking prompt starts missing.

Rewritten conversation history: Summarizing, truncating, or compacting earlier turns changes the prefix. Appending text inside an existing message can also remove a boundary that implicit lookup previously used. Preserve earlier messages when appropriate, and compare total cost after compaction because a shorter prompt may save money despite reduced reuse.

Prefixes below the minimum: Supported minimum lengths range from 128 tokens on some Groq models to 4,096 tokens on several Claude and Gemini models. A shorter prompt may run normally while producing no cache reads. On Anthropic, both cache_creation_input_tokens and cache_read_input_tokens reading zero is the signal.

Expired entries: A follow-up outside the cache lifetime may need a fresh write. Anthropic measures its lifetime from the start of the request that writes or reads the entry, so a response that streams for four minutes leaves roughly one minute of a five-minute lifetime for the follow-up to begin.

Cold parallel requests: On Anthropic, a new entry becomes available after the first response begins. Concurrent requests sent before that point may all miss a cold prefix, so wait until the entry is available before sending requests intended to reuse it.

Routing or isolation boundaries: Caches are scoped by provider-defined organization, workspace, and regional boundaries. On OpenAI models before GPT-5.6, a stable prompt_cache_key helps related traffic reach a machine holding the prefix. Routing is automatic on newer models, where keys instead provide separate cache accounting.

Partial reuse on Gemini 4: For Gemini 4 Argon and newer models, a new request must add at least 2,050 tokens to reuse the full previous request. With a smaller addition, Gemini reprocesses the previous request's last 2,050 tokens at the standard input rate; a request can therefore report partial reuse.

Compare affected requests with OpenAI prompt cache diagnostics or Anthropic cache diagnostics to locate the boundary or content change responsible for a miss.

Prompt caching controls and cache lifetime by provider

Caching may be automatic, explicitly configured, or tied to a separate cache object. Confirm the model and API before applying a provider's settings, because support can differ between model generations and endpoints.

Automatic and explicit prompt caching

OpenAI: Supported models cache prompts by default. On GPT-5.6 and later, prompt_cache_options.mode selects implicit or explicit-only caching, and prompt_cache_breakpoint marks a chosen content boundary. A request can create up to four writes, including the implicit breakpoint when enabled. Earlier models support implicit caching only.

Anthropic: On the Messages API, prompt caching is opt-in through cache_control. A top-level field enables automatic caching as the conversation grows, while block-level markers define explicit boundaries. The request supports up to four breakpoints, allowing stable tools or instructions to have a boundary separate from changing conversation content. If four explicit block-level breakpoints already exist, the API returns a 400 because no slot remains for automatic caching.

Google Gemini: Gemini context caching is implicit by default on Gemini 2.5 and newer models. Explicit cache objects are available on supported Gemini 3 and 2.5 models through generateContent. The Interactions API and Gemini 4 models support implicit caching only.

Groq: Prompt caching is automatic on supported GPT-OSS 20B, GPT-OSS 120B, and GPT-OSS-Safeguard 20B models. Groq exposes no manual cache controls, and caching cannot be disabled.

Cache pre-warming: Anthropic accepts compatible max_tokens: 0 requests that write configured prefixes without generating output. OpenAI's GPT-5.6 and later models offer prompt_cache_options.prewarm in the Responses API. Preparing the prefix can reduce input-processing delay on the first user request, but both providers charge for writes, so choose prefixes likely to be reused before coverage expires. Anthropic's zero-output pre-warming has configuration restrictions and cannot run inside a batch request.

Minimum prefix length and cache lifetime

The following comparison reflects provider guidance checked on November 13, 2026. Minimum lengths are model-specific, and lifetime settings describe different retention or billing guarantees across providers.

ProviderMinimum prefixDefault lifetimeLonger-lifetime option
OpenAI GPT-5.6 and later1,024 visible input tokens30-minute minimum coverage12-hour minimum in the Responses API
Anthropic512 to 4,096 tokens5 minutes; reads refresh the entry1 hour
Google Gemini2,048 to 4,096 tokensImplicit lifetime is not publishedConfigurable explicit cache objects on supported models and APIs
Groq128 to 1,024 tokensExpires after 2 hours without useNo configurable extension

OpenAI's minimum lifetime governs eligibility for cache-read pricing and does not guarantee physical KV reuse or lower latency on every request. On older OpenAI models, retention differs: GPT-5.5 and GPT-5.5 Pro use 24h, while other earlier models support in_memory or 24h where documented.

Prompt caching pricing: cache reads, cache writes, and uncached input

A request can contain ordinary input, cache reads, cache writes, and generated output. Calculate each category at its own rate before deciding whether caching reduced spending.

How each token type is billed

Uncached input: Tokens outside the reused prefix bill at the standard input rate unless they are being written into a new cache entry.

Cache writes: A provider may charge a premium for storing eligible input. On OpenAI's GPT-5.6 and later models, the write rate replaces the ordinary input rate for those tokens, so counting both rates would double-charge them in a cost calculation.

Cache reads: Reused tokens receive the model's cached-input rate. The multiplier depends on the model, so a provider-wide percentage is insufficient for an exact cost report.

Storage and output: Gemini explicit caches can add token-storage charges over time. Output remains billed at the applicable output rate, including any service-tier discount that applies independently of prompt caching.

The table summarizes input-price multipliers for the standard service tier. Multiply each factor by the selected model's base input price.

ProviderCache-write multiplierCache-read multiplierAdditional consideration
OpenAI API pricing1.25x for 30 minutes; 2.5x for 12 hours0.1x; 0.05x on GPT-6.1 Sol and TerraEarlier models have no additional write premium
Claude API pricing1.25x for 5 minutes; 2x for 1 hour0.1x; 0.05x on Opus 5.5 and Sonnet 5.5; 0.025x on Fable 5.1 and Mythos 5.1Confirm the exact model's cached-input rate
Gemini API pricingNo separate premium for implicit writes0.1x on many models; 0.05x on Gemini 4 ArgonExplicit cache objects add hourly storage charges where supported
Groq models and pricingNo write premium0.5xThe caching discount does not stack with the batch discount

Use the selected model's current prices when forecasting spend, and preserve the rates applied to historical requests. Gemini lists scheduled price changes, so a cached token count needs a model, service tier, and billing date to translate into dollars.

Break-even point for cache writes

The following examples compare only the repeated prefix's input cost. Include new input, generated output, storage, and later rewrites when calculating the application's complete bill.

One read covers a short-lifetime write: With a 1.25x write and 0.1x reads, one write followed by one read costs 1.35 times the prefix's uncached price, compared with 2 times for two uncached requests. Across ten requests, one write and nine reads cost 2.15 times, compared with 10 times uncached.

Higher write rates require additional reads: A 2x write followed by one 0.1x read costs 2.1 times the uncached prefix price, exceeding the 2 times for two uncached requests. Two reads bring the total to 2.2 times across three requests, compared with 3 times uncached. A 2.5x write plus two reads costs 2.7 times, so it also breaks even after the second read under these assumptions.

Lifetime choice follows reuse intervals: A five-minute Anthropic cache fits frequently reused prompts because reads refresh the entry. An hour-long cache can serve a conversation or reference prefix reused after a longer pause, provided the additional reads offset its higher write cost. OpenAI's longer minimum coverage requires the same comparison against likely rewrites after shorter coverage expires.

Crossing the minimum length needs measurement: Adding useful instructions or examples can make a short prefix cacheable, but the extra tokens change both the task and its cost. In OpenAI's example with a 1,024-token minimum, a 1.25x write, 0.1x reads, and ten total requests, expanding a prefix of at least 221 tokens can reduce prefix-input cost. Recheck answer quality and expected reuse before adopting the longer prompt.

Workloads that benefit from prompt caching

Caching is most useful when a substantial prefix repeats within the provider's reuse window. The same prompt structure can produce different savings under interactive traffic, sparse requests, and parallel batch execution.

Multi-turn chat: Preserving conversation history lets later turns reuse earlier context. Rewriting the history or extending an old message can reduce that benefit, even when the conversation remains logically similar.

Agents with large tool sets: Repeated instructions and tool schemas create stable prefixes across agent steps. New tool results can follow the shared content, provided the application preserves the earlier context and the provider can find its cache boundaries. OpenAI reports a token cache-hit rate above 90% for one example agent deployment; the rate depends on the application's context and usage.

Document question answering: Questions about the same contract, manual, or codebase can share the reference material. Savings depend on document length, request timing, and any provider-specific limits on reusing the previous request.

LLM-as-a-judge evaluation: A fixed judge rubric and labeled examples can be cached ahead of the interaction being scored, which keeps the repeated evaluation instructions reusable while each candidate answer remains new input. OpenAI's single-turn judge example reported approximately 70% token cache hits.

Short prompts, frequently changing prefixes, and infrequent reuse may produce little benefit. With write premiums, entries that expire unread can raise input costs, so test expected traffic patterns before treating a high theoretical cacheable share as a saving.

Prompt caching vs KV caching vs response caching vs semantic caching

Four mechanisms reuse different parts of an LLM application, and which part is reused determines whether the model generates an answer and which costs remain on a hit.

MechanismWhat is reusedReuse scopeWork remaining
KV cachingKey and value vectors for earlier tokensTypically within an inference runThe model continues processing and generating tokens
Prompt cachingIntermediate states for a shared prefixAcross eligible requests with matching prefixesProcess the new suffix and generate a response
Gateway response cachingA complete stored responseAcross requests matching the cache keyReturn the stored response without another generation call
Semantic cachingA stored response for a similar requestAcross requests passing a similarity thresholdMatch the request and return the stored response; embedding work may still incur cost

Prompt-cache reads reduce the price of reused input but leave new input and output charges in place. Response-cache hits skip the generation call, although stored answers can become stale. Semantic matching adds the risk of returning an answer for a related but different request when the similarity threshold is too permissive.

The Braintrust Gateway uses exact-match response caching. Its default auto mode caches supported requests that set temperature to 0 or provide a seed. A Gateway hit returns the stored response without contacting the model provider, so provider prompt caching does not run for that request.

When the Gateway forwards a request, supported provider caching parameters can still apply; they retain their provider-specific meaning. Set x-bt-use-cache to never when measuring provider prompt caching so each request reaches the provider for a fresh generation.

Also read: LLM gateway caching and response reuse

Measuring prompt caching savings with cached-token usage and latency

Provider usage fields describe cache activity, but their input-token totals follow different conventions. Normalize the counts before comparing workloads or calculating savings.

Reading cached-token usage fields

OpenAI: In the Responses API, input_tokens includes cache reads and writes. input_tokens_details.cached_tokens reports billed reads, while GPT-5.6 and later models report writes in input_tokens_details.cache_write_tokens. Ordinary input equals the total minus reads and writes. Where both write lifetimes are used, price their separate counts at the corresponding rates. Chat Completions reports reads under prompt_tokens_details.cached_tokens.

Anthropic: cache_read_input_tokens reports reads, cache_creation_input_tokens reports writes, and input_tokens reports the remaining input after the cache boundary. Total input is the sum of all three. A request with 100,000 read tokens, no writes, and a 50-token suffix therefore contains 100,050 input tokens despite reporting input_tokens as 50.

Google Gemini: In the Interactions API's Python and JavaScript SDKs, usage.total_cached_tokens reports cache-hit tokens. Other API surfaces use their own usage schemas, so map the fields for the endpoint actually used by the application.

Groq: prompt_tokens includes all input, and prompt_tokens_details.cached_tokens identifies the cached portion. In Groq's example, 4,608 of 4,641 input tokens came from cache, producing a token cache-hit rate of approximately 99.3%.

Adding OpenAI cached tokens to its input total counts the same tokens twice. Treating Anthropic's input_tokens as the total omits the cached portion, which can account for most of a long prompt.

Also read: Track LLM token usage across prompts, outputs, and agent steps

Cache hit rate and realized cost

Token cache-hit rate is the number of cache-read tokens divided by total input tokens. For a workload, divide the sum of reads by the sum of input tokens over the same population and period. Averaging request-level percentages gives a short prompt the same weight as a much longer one and can misrepresent overall reuse.

A high read rate says nothing about the cost of creating entries. Count writes at their lifetime-specific rates, reads at the cached-input rate, ordinary input at the base rate, and generated output at the output rate. Add explicit storage charges where applicable, then compare equivalent workloads or task volumes.

Keep full-response cache hits separate from provider prompt-cache reads. A request answered by the Gateway has not made a new provider call, even when the stored response payload contains usage figures from the original generation.

Latency signals and time to first token

Prompt caching reduces input-processing work, so time to first token is the main latency signal to examine for streaming requests. Total response duration also includes generation, and a long answer can take substantial time after the first token appears.

Compare cache-read and cache-miss requests using the same model and reasoning settings on the same service tier, with similar prompt and output lengths. Keep queueing, network conditions, and streaming configuration in view so a difference in total duration is not attributed entirely to caching.

Verify the latency benefit with observed TTFT rather than inferring it from cached-token counts. In Braintrust, confirm how the integration records time_to_first_token, since some wrappers measure the first stream event before visible text appears.

Tracking prompt cache performance with Braintrust

Cache optimization needs to connect a request's usage, its prompt version, and the result delivered to the user. Braintrust keeps the available metrics on the same traces so engineers and product reviewers can investigate higher spending without losing the context behind each call.

Step 1. Normalize cache usage on LLM spans

Supported integrations and SDK wrappers map provider usage into shared metrics. prompt_tokens includes all input, prompt_cached_tokens records reads, and prompt_cache_creation_tokens records writes. The cache counters are subsets of total input, so adding them to prompt_tokens would inflate usage.

For custom logging, map provider usage to normalized cache metrics and report completion_tokens as well. Braintrust calculates the cache hit rate as prompt_cached_tokens / prompt_tokens, and the cache-hit label also requires completion_tokens; raw provider cache fields alone can leave the UI showing 0% even when the provider reports a hit.

Step 2. Inspect reuse within each trace

Use cache metrics in the trace view to locate agent steps with repeated writes or little reuse. The Timeline reports a hit rate on the root span and on every LLM call beneath it, so a cold first request and the warm continuations that follow it are distinguishable without opening each span.

Braintrust Timeline showing an 85% cache hit rate across LLM spans with per-span token and cost detail

To compare reuse across calls directly, switch the Timeline to Cached tokens view from its options menu. That view isolates LLM spans, flattens the hierarchy, and scales each bar by cached read tokens, with stacked segments for uncached input, cached read, and cache write.

Review writes, reads, and estimated cost together. Cost estimates depend on the captured metrics and available model and provider pricing, so compare them with provider billing when validating a new model or cache configuration. Investigate missing usage data before concluding that caching saved money.

Step 3. Compare prompt versions over time

Link prompt-version metadata to traces and record the exact version used for each call. A prompt's stable name or ID identifies the prompt, while a version identifier distinguishes revisions. Keep the model and provider alongside the version so a price or model change is visible in the comparison.

Use dashboard filters and groupings to compare cached tokens, cost, and streaming latency across the relevant workload, with custom dashboards and charts available on paid plans. SQL queries can aggregate metrics.prompt_cached_tokens with estimated_cost() by model and day.

Braintrust dashboard showing span volume, latency percentiles, and a cost breakdown separating cached and uncached prompt cost

Product managers can use Loop, Braintrust's AI assistant, to ask cost and usage questions in plain language, reducing the engineering work needed for each investigation.

Step 4. Validate the prompt change before rollout

A reordered prefix or additional examples can change answer quality as well as cache behavior. Evaluate cache-friendly prompt changes against the current version on the same cases, checking quality alongside cost and latency before rollout.

For requests routed through the Gateway, use x-bt-cached to distinguish response-cache HIT and MISS results. Gateway logging through x-bt-parent can place requests in traces, but provider-cache analysis should focus on calls that actually reached the provider. Keep the accepted prompt version as the reference for monitoring subsequent cache and spending changes.

Braintrust's free Starter plan includes 1 GB of processed data per month and unlimited users, with 14-day retention. Engineering and product can review the same initial cache measurements before expanding the setup.

Trace prompt cache reads, writes, and cost with Braintrust for free.

FAQs about prompt caching (2026)

Does prompt caching change model outputs?

The model generates a new response on every prompt-cache hit, so normal sampling and inference variation can still produce different answers. When instructions or examples are changed to improve cache reuse, Braintrust experiment comparisons reveal whether the revised prompt introduces quality regressions.

Is prompt caching the same as caching LLM responses?

Prompt caching reuses processed input and leaves generation in place. Response caching returns a previously generated answer, skipping the generation call and its token charges. Use response caching only when the stored answer remains valid for the new request.

How long does a prompt cache last?

Lifetime depends on the provider, model, and selected cache mode. Some reads refresh coverage, but OpenAI's 30-minute reads do not restart a previously purchased 12-hour window. Match the setting to expected reuse intervals and distinguish minimum pricing coverage from the provider's maximum retention period.

Do cached tokens count toward rate limits?

OpenAI counts cached input toward tokens-per-minute limits. Anthropic excludes cache reads from input-token limits on most models, with Haiku 3.5 the documented exception, while writes and ordinary input still count. Groq subtracts cached tokens after processing, so parallel requests can reach a limit before the adjustment is applied.

Why are cached tokens zero when the system prompt hasn't changed?

A stable system prompt can still miss if the complete prefix changes, the cache expires, or the request fails the provider's caching requirements. Braintrust's LLM tracing records the request and available usage metrics together, helping distinguish a provider miss from missing or incorrectly mapped cache fields.

Does prompt caching work with batch processing?

Supported batch APIs can reuse prefixes, but execution order and discount rules differ. Anthropic offers best-effort batch cache hits and combines caching multipliers with batch pricing. Groq keeps caching active without stacking its caching discount on the batch discount, and Gemini publishes separate cached-input batch rates.

Is prompt caching safe for sensitive data?

Prompt caching can suit sensitive workloads when the selected model's retention and isolation policies meet the application's requirements. OpenAI may retain encrypted KV state for up to 24 hours; Anthropic holds cache state in memory for its configured lifetime and offers Zero Data Retention eligibility with model exclusions. Apply customer-specific cache accounting or provider isolation controls consistently with the application's tenant boundaries.

Share

Trace everything