What is model routing? How LLM routers choose the right model

---

What is model routing?

Model routing applies a selection policy to each application request, using the request and its context to determine which model, provider endpoint, or combination should handle it.

The policy defines a candidate pool of available models and providers. Eligibility constraints then remove options that cannot meet requirements such as context length, tool support, or an approved processing region. A selection rule chooses among the remaining candidates, and a fallback order specifies subsequent attempts when the selected endpoint encounters a retryable failure. The resulting route can also specify generation settings.

Where routing logic lives: Application code can make routing decisions using product context such as the feature, tenant, or user tier. A gateway or proxy can apply a shared policy across requests passing through it, along with centralized credentials, logging, and failover. A managed router endpoint delegates the selection mechanism to a service, and how much configuration and control you keep over the candidate pool depends on that service.

Routers, gateways, and unified APIs: A unified LLM API standardizes request and response formats across providers, and a gateway adds controls such as credential management, caching, and logging. A router chooses the destination, and some products combine all three functions behind one endpoint.

Model routing strategies

Routing strategies use different evidence to make a selection. Some inspect the incoming request, others use recent operational measurements, and quality-based policies draw on results from completed responses.

Static rule-based routing

Static rules map known request attributes to configured routes. A support-chat feature might use one model, a bulk-tagging job another, and a tenant with residency requirements an approved regional endpoint.

Rules can match features, tenants, user tiers, endpoints, or metadata, making the selection logic straightforward to inspect. Assign an owner to review mappings when models, prices, or application requirements change.

Availability and failover routing

Availability routing uses health checks, recent failures, or capacity signals to avoid endpoints that cannot serve a request. Failover retries eligible failures, such as timeouts, rate limits, or server errors, through a configured sequence of alternatives.

Depending on the implementation, unhealthy endpoints may enter a cooldown period before receiving traffic again. Fallbacks can use another provider serving the same model or switch models, with model-changing chains requiring evaluation against the original task requirements.

Latency-based routing

Latency-based routing selects candidates using recent response-time or throughput measurements. A rolling window keeps decisions tied to current serving conditions, and percentiles reveal variation an average can hide.

For streaming interfaces, time to first token measures how quickly users begin receiving a response. Total generation time and tokens per second measure work that depends on a complete answer, including batch processing and sequential agent steps.

Cost-based routing

Cost-based routing chooses among eligible candidates using estimated call costs or configured price ceilings. Estimates should account for expected input and output usage, cache behavior, and the applicable service tier.

Assess the resulting policy using cost per successful request, calculated as total workflow spending divided by responses that pass the task's quality checks. Include routing, retries, and escalation calls so the comparison covers the complete path to an acceptable answer.

Task-type routing

Task-type routing classifies the requested job, such as extraction, summarization, classification, or code generation, then assigns it to an approved model. Labels can come from keyword rules, similarity to embedded examples, a small classifier, or an LLM.

Test the classifier against labeled examples separately from testing the selected models. A model can perform well on its assigned task and still receive unsuitable requests when the classifier mislabels them.

Complexity-based routing

Complexity-based routing predicts how demanding a request will be and selects a corresponding model tier. Heuristics, trained predictors, or LLM classifiers can estimate difficulty, with configurable thresholds controlling how often requests reach more capable models.

Validate predictions against the application's actual requirements, including exact citations, structured fields, or specialized knowledge. For managed routing services, also check restrictions on the candidate pool and the context limits available across its models.

Quality-based routing

Quality-based routing uses measured or predicted response quality to guide model selection. One practical implementation evaluates candidate models on representative requests from each traffic segment, using task-specific checks and acceptance thresholds to establish which models can serve that segment.

Production scoring then provides evidence to review those assignments as traffic and model behavior change. The policy's reliability depends on whether the datasets represent incoming requests and whether the scorers reflect acceptable task outcomes.

Model routing strategies compared

Apply hard eligibility constraints first, then use rules, classification, or quality evidence to determine suitable model options. Cost, latency, and availability can guide selection within that approved pool and among its provider endpoints, with failover handling eligible execution errors. Define the precedence between policies so each decision has an identifiable rule or signal behind it.

Also read: How to cut LLM costs with model routing without hurting quality

Model selection vs. provider selection

Model selection determines which model answers a request, while provider selection determines which endpoint serves it. Depending on availability, the same model may be accessible through its developer's API, a cloud service, or independent inference hosts. Separating these decisions helps teams assess changes in model capability and serving behavior.

Provider changes need output checks: A request can complete successfully even when the endpoint does not honor every requested setting. Some routing services allow unsupported parameters to be ignored unless strict compatibility filtering is enabled. For applications that depend on structured output or specific generation settings, checking only the HTTP status can miss behavioral changes.

Logging both decisions: Record the model version and provider separately so quality results can be compared across endpoints. A decline concentrated on one provider gives investigators a starting point, but traffic differences, prompt changes, and generation settings can also explain the pattern. Replaying the same requests while holding those factors constant provides stronger evidence that the provider change caused the regression.

The LLM routing request lifecycle

A routed request typically passes through the following stages. Classification and fallback appear only when the policy calls for them.

1. Request intake: The router receives the request alongside application context such as the feature, tenant, user tier, session ID, prompt version, and routing hints. This metadata allows requests with similar prompts to follow different policies when their application requirements differ.

2. Classification: Policies based on task type or difficulty assign a label or score to the request. Keyword rules require little processing, but an embedding-based or LLM classifier can add an inference call. Include that overhead when measuring the route's total cost and latency.

3. Eligibility filtering: The router excludes candidates that cannot meet requirements for context length, input modality, tool calling, structured output, data residency, or budget. These constraints apply to fallback destinations as well as the first choice.

4. Policy evaluation: The selection rule chooses a model and provider from the eligible candidates, along with generation settings and an ordered fallback list. Some implementations select the model first, then choose among providers serving it.

5. Execution: The router sends the request to the selected endpoint and streams or returns the response. When the application checks the output before delivery, those checks also add to response time.

6. Fallback: A retryable rate limit, server error, or timeout can trigger another attempt according to the configured retry limits and fallback order. Some routers also support context-window fallbacks that send an oversized request to a model with greater capacity.

A completed call can still produce an unusable answer. Quality-based escalation requires an output check and an application rule defining the next attempt. Schema validation can detect invalid structure, but assessing whether the answer is correct requires task-specific checks or scoring.

7. Logging the decision: Record the model version, provider, selection rule, classification result, and fallback events alongside latency, token usage, and cost for each attempt. Linking these records to the request, prompt version, and generation settings supports investigation and later replay.

Switching models during a conversation can change tone, formatting, and tool behavior. Changing routes can also reduce prompt-cache reuse, depending on the provider's cache rules. Session stickiness keeps a conversation on the same model and provider when available, reducing route changes without guaranteeing identical behavior across turns.

Agents can select models separately for individual steps, such as using a stronger model for planning and a smaller one for formatting tool arguments. Each step needs its own routing record so you can trace an unsuccessful task to the decision and response that contributed to it.

Data needed for a defensible routing policy

A routing policy needs evidence that its candidate models can meet the application's quality, latency, and cost requirements on representative requests. Collecting comparable measurements for each model and provider lets you assess whether a proposed route improves the application.

Representative traffic: Sample production logs by feature, task type, and tenant to establish which requests the router will handle and in what proportions. Include infrequent, consequential cases alongside common requests so the evaluation reflects the application's operating requirements.

Task-specific evaluations: Build datasets from the sampled traffic and assess each candidate using code checks, LLM judges, and human review as appropriate. Results should show whether each model meets the acceptance criteria for the segments it would serve.

Latency distributions: Measure response times separately for each model and provider, including p50, p95, and p99, plus time to first token for streaming. Include classification and retry delays when assessing whether the complete routed request meets the application's response-time targets.

Token costs: Calculate cost per request and per successful request for each segment, including classification, retries, and escalation. Apply the relevant prices to provider-reported usage carefully, since cached input and reasoning tokens can be subsets of broader totals and must not be counted twice.

Failure rates: Track errors, timeouts, rate limits, invalid output schemas, and inappropriate refusals by model and provider. Separate these failure types so a low overall error rate does not conceal a recurring problem with a required capability.

Segment-level evidence: Review quality, latency, cost, and failures using the same segments the policy routes on. An acceptable overall pass rate can hide poor performance on a low-volume segment, such as refund disputes, especially when failures there have greater consequences.

Evidence freshness: Record the measurement window, model and prompt versions, and pricing assumptions so results remain interpretable. Changes to providers, prices, or the application's traffic mix should trigger a review of the affected routes and new measurements where needed.

Validating and rolling out a routing policy

How to test a cheaper model in Braintrust

Before routing traffic to a cheaper model, use Braintrust to test whether it can meet the feature's quality requirements. The workflow connects production traces, datasets, prompt iteration, and experiment comparisons so the routing decision rests on inspectable results rather than a few convincing responses.

1. Create a dataset from the feature's traces. In Braintrust's Logs view, filter to the feature you want to reroute. Select representative traces, then choose Add to > Add to dataset. Promoting traces from logs preserves a path from a test case back to the production behavior that prompted it.

Include common requests and consequential edge cases, and retain metadata for the segments the policy will route on, such as task type or customer tier. Review expected answers before using them as references: the current model's output is not automatically correct. Keep separate development and held-out validation sets so prompt tuning does not consume the examples used for the final decision.

2. Run the current and cheaper models as experiments. In a Braintrust playground, attach the dataset and add two prompt tasks: the current model as the baseline and the cheaper model as the candidate. Start with the same prompt and comparable generation settings to isolate the model change. Attach the same task-specific scorers to both, run the tasks, and save each configuration with + Experiment to retain the results.

Use code checks for requirements such as valid schemas and required fields, and an LLM judge or human review for qualities those checks cannot establish. Before trusting a judge, compare its judgments with human labels on examples from each important segment, inspect disagreements, and refine the rubric. Run each case more than once to expose inconsistent answers; repeated runs measure variability but do not replace a broader sample of distinct requests.

3. Tune the cheaper model's prompt in the playground. The production prompt may have been tuned for the current model. A weak first result does not establish the cheaper model's ceiling. On the development set, adjust instructions, examples, or output requirements one change at a time, using the playground's output diffs and scores to inspect the effect. Save promising configurations as new experiments rather than overwriting the baseline. Then evaluate the selected prompt on the untouched validation set. This separates the initial model-only comparison from the practical decision about deploying a tuned model-and-prompt pair.

4. Read regressions for each segment in the comparison view. Open the candidate experiment and select the current-model experiment as its baseline. Braintrust's experiment comparison aligns test cases and shows score deltas. Filter by segment metadata, surface regressions, and enable Diff to inspect changes in outputs. Use the Summary table to compare recorded scores, cost, and latency, then return to individual cases to understand what the averages hide.

Set the acceptable quality loss before interpreting the result. A change from 94% to 90% on only 50 examples is not enough on its own to establish a reliable regression—or equivalence. Choose sample sizes around the smallest quality loss that matters, with adequate coverage for each required segment. Calculate confidence intervals for paired score differences, for example with a paired bootstrap over distinct test cases; keep repeated trials grouped with their case. If the interval still includes an unacceptable loss, collect more evidence rather than declaring the models interchangeable.

Approve the cheaper route only for segments where the tuned candidate meets the quality threshold with sufficient evidence and delivers worthwhile savings. Include classification, retries, and escalation in cost per successful request and end-to-end latency. If the candidate passes routine requests but fails refund disputes, keep disputes on the stronger model and validate that routing rule before rollout.

Gradual rollout and rollback triggers

Shadow testing sends live requests to the candidate policy in parallel with the current setup, retaining candidate responses for evaluation without showing them to users. Running both paths adds inference cost, but reveals how the policy handles current inputs.

A canary rollout then exposes a small share of eligible traffic to the candidate, limiting the number of users affected as production evidence accumulates. An A/B test can compare user or business outcomes between the candidate and current policy. Application feature flags or gateway traffic controls determine which requests reach each configuration.

Define rollback conditions before exposure begins, such as a segment's quality score staying below its minimum for a specified window, an error rate exceeding its ceiling, or p95 latency crossing its target. Assign an owner and verify that the previous policy can be restored. Alerts notify the responsible team or an automation service, and the application or routing controls restore the previous policy.

Quality drift monitoring in production

Compare production behavior with the validated baseline to distinguish a deteriorating route from a change in the requests reaching it.

• Route distribution: Investigate unexpected changes in traffic share by checking the request mix, classifier decisions, and fallback frequency before attributing the shift to a single cause.
• Per-route quality: Track scores by route and segment, allowing enough scored requests in each monitoring window to interpret changes reliably.
• Provider changes: Reevaluate affected workloads when serving behavior changes, and update cost comparisons when provider prices change.
• New model releases: Treat a release as a prompt to test the candidate against current requests, and replace an established route only when measured results justify it.

Add confirmed production failures to the replay dataset so subsequent policy changes are tested against the problems users encountered.

Also read: How to build a prompt CI/CD pipeline and How to set up quality, cost, and latency alerts for AI agents

When model routing is worth it

Consider routing when one model or endpoint cannot meet the workload's requirements efficiently. The expected benefit needs to justify the engineering time spent on selection logic, evaluation, and monitoring.

Start with failover. When availability is the main concern, test a backup endpoint for the existing model before introducing task-based model selection. Verify compatibility, output quality, and recovery under simulated failures, then expand model selection when traffic differences and evaluation results justify it.

Model routing with Braintrust

Braintrust connects model evaluations with production traces so you can assess routing changes before and after release. The application router selects the model, and the Braintrust Gateway sends each request to a configured provider.

Compare candidate models on real requests

Promote production traces into datasets to build evaluation sets for each routing segment. In playgrounds, run candidate models with the same built-in scorers, LLM judges, or custom scoring logic. Diff mode highlights changes in outputs, scores, timing, and token usage, and runs can be saved as experiments.

Experiment comparison identifies cases that improved or regressed against the baseline. Its Summary table grades overall results using scores and metrics covering latency, cost, errors, and load.

The Braintrust eval GitHub Action runs suites on pull requests and posts results as comments, and bt eval runs the same suites in other CI systems. Custom reporters define score-based failure conditions that required CI checks can enforce before merge.

Collaborators can also use Loop to investigate traces, create datasets, and run evaluations through plain-language requests.

Configure provider selection and failover

The Braintrust Gateway exposes a unified API. Provider credentials are configured at the organization or project level, and provider eligibility depends on model support, configured restrictions, and caller permissions.

Use x-bt-endpoint-name to select a specific configured endpoint and x-bt-fallback-providers to specify an ordered, comma-separated list of backup provider credentials for provider failover. The Gateway starts with the provider it would normally select, then works through that list, retrying eligible provider failures such as unavailability, rate limits, and server errors. It does not retry Gateway authentication errors, Gateway validation errors, or provider client errors other than rate limits.

The hosted Gateway is in public preview and free to use, with no SLA, so be prepared for changes to behavior and configuration. You still pay your own providers for inference, since the Gateway calls them with your configured credentials.

Connect routing outcomes to traces

The x-bt-used-endpoint response header identifies the provider credential that completed a request. After successful failover, x-bt-failover-from and x-bt-failover-to identify the original and final model/provider pairs.

Set x-bt-parent with the parent span's exported context to log Gateway calls within the application trace, connecting latency, token usage, and estimated cost with scores attached later. The Gateway returns x-bt-span-id for that logged span, which you can use to attach scores or feedback after the request completes.

Monitor each route after rollout

Online scoring evaluates production traces asynchronously, outside the user-facing response path, with filters and a sampling rate configured per rule. Record route and segment metadata on the spans used by each chart, then use dashboards to compare quality, cost, and latency. Every project includes a read-only Cost and quality dashboard; clone it or create your own to add custom charts, which are a Pro and Enterprise feature.

Time window alerts notify Slack, PagerDuty, or a webhook when an aggregate crosses a threshold. A breached threshold can trigger the rollback conditions defined during rollout.

Notion uses Braintrust to compare regression and frontier-model performance, identify strengths for specific use cases, and deploy new frontier models in under 24 hours.

Braintrust's free Starter plan includes 10,000 scores per month, plus unlimited datasets, playgrounds, and experiments to evaluate candidate models and monitor route quality.

Start free and evaluate your model routes with Braintrust.

FAQs about model routing (2026)

What is the difference between model routing and load balancing?

Model routing determines which model or provider should handle a request. Load balancing distributes traffic across a serving pool to manage capacity and availability. The two combine when routing selects a model and load balancing spreads that model's requests across eligible endpoints.

Does an LLM router add latency?

Routing introduces processing overhead, but total response time can still decrease when the selected endpoint completes the task faster or avoids congestion. Compare the complete request path with the current setup, including classification, network transit, and any additional attempts.

What is semantic routing?

Semantic routing selects a destination using the meaning of an input. A common implementation compares the request's embedding with examples assigned to each route. Similarity thresholds and an explicit fallback handle requests with no sufficiently strong match.

What is the difference between a model router and a model cascade?

A cascade is a routing strategy that uses an initial model's response to decide whether to call another model. Its value depends on how reliably the output check identifies responses needing another attempt, and how often escalation adds enough benefit to justify the extra call.

What should an AI gateway support for conditional routing rules?

Look for explicit rule precedence, constraints enforced across fallback destinations, and limits on retry time and cost. Traffic-splitting controls support staged releases, and decision logs should identify the policy version, matched rule, and final destination so unexpected routing can be investigated.