An MCP server can pass every protocol and handler test and still break down once an agent has to pick a tool, fill in valid arguments, and work through errors on the way to finishing a task. A scorer that reads only the final response misses silent failures and duplicate records left behind by a retry.
An MCP eval runs an agent against realistic tasks, records every model turn and tool call as a trace, and scores both the full trajectory and the system state the run leaves behind. Repeating each task across trials turns those records into a reliability number covering tool choice, argument construction, and multi-step completion.
This guide explains how MCP evals differ from protocol tests and public benchmarks, what they should measure, and when teams should run them. Braintrust connects traced MCP behavior to repeatable experiments and release requirements, letting teams compare server changes and turn production failures into regression cases. Start evaluating your MCP server with Braintrust →
What is an MCP eval
An MCP eval is a repeatable test that measures how reliably an AI agent uses an MCP server. The eval presents the agent with realistic tasks, captures every model turn and tool call in a trace, and applies scorers to the recorded behavior. Each task runs several times so the results show a success rate across trials rather than a one-time pass or failure.
A request sent through MCP Inspector or curl confirms that the server accepts known arguments and returns the expected response. An MCP eval tests whether the model can interpret the tool descriptions, choose the correct tool for an unfamiliar request, construct valid arguments, and use the server response to complete the task.
Six components make up an MCP eval:
- Task set: Requests written the way users phrase them, together with control cases where the correct response does not require a tool.
- Agent and client: The selected model, system instructions, and MCP client configuration. Pinning each input keeps later results comparable with the baseline.
- Server under test: A specific server version with a fixed tool list, schemas, and permission scope.
- Trace: The complete sequence of model turns, tool calls, arguments, server responses, and errors from one run.
- Scorers: Deterministic checks and model-based judges that assess the recorded behavior and resulting system state.
- Repeated trials: Multiple runs of the same task that reveal how consistently the agent completes it.
A repeatable MCP eval ties each task-level pass rate to the agent behavior and system state behind it.
The eval harness acts as an MCP client, so it should use the same transport, authentication path, and exposed tools as the client used in production. A harness that skips OAuth or substitutes a shortened tool list is measuring a different configuration, and its scores say nothing about how reliably agents will use the production server.
What MCP evals measure
A single pass-or-fail score shows a run failed but does not identify the cause. Effective agent evaluation separates tool use, arguments, execution, and outcome into distinct measurements. Each low score can then point to a specific part of the server or agent configuration.
- Tool necessity: A necessity check compares the agent's decision to call a tool against what the task required, which catches unnecessary calls that add latency and cost without moving the task forward.
- Tool selection: The chosen tool should match the user's request even when the server exposes similar options. If the agent confuses search_orders with list_orders, their names or descriptions may not distinguish the functions clearly enough.
- Argument correctness: Each call must conform to the tool's input schema, include the required values from the user request, and follow enum or format constraints. Dates, identifiers, and nested objects are common sources of invalid arguments.
- Sequencing: Multi-step tasks depend on calls occurring in the correct order and data passing from earlier responses into later requests. A valid individual call can still fail the task if it occurs too early or lacks information from a previous step.
- Error recovery: A server error, empty result, or permission denial tests how the agent responds when the expected path fails. Repeating the same invalid call or treating a failed operation as successful should reduce the score.
- Task success: The final measurement checks the system state after the run. It catches cases where the agent reports success even though the underlying operation failed or remained incomplete.
Separate scores make failures easier to diagnose without reducing the entire run to one result. An agent can select the correct tool and submit valid arguments, then stop short of the remaining calls and still fail the task.
How the three MCP primitives get evaluated
An MCP server can expose tools, resources, and prompts. MCP assigns control of each primitive to a different participant, so the evaluation criteria should reflect who decides when and how it is used.
- Tools: The model chooses when to invoke a tool. Evals should examine the selected operation, submitted arguments, call sequence, recovery behavior, and final outcome. Tools that modify data also need a state check to confirm that the requested change occurred.
- Resources: The application retrieves resources and adds their contents to the model context. An eval should confirm that the application fetched the relevant resource and that its contents contributed to the response. Successful retrieval alone does not establish that the agent used the information correctly.
- Prompts: The user selects a prompt explicitly and supplies its arguments. Evals should test whether the completed template produces consistent results across supported models and clients.
MCP testing vs MCP evals
A reliable MCP testing strategy combines server and protocol tests with MCP evals. Server tests validate implementation behavior against known requirements, and MCP evals measure agent performance across realistic tasks.
| Comparison point | Server and protocol tests | MCP evals |
|---|---|---|
| Question answered | Does the server return the expected response to a known request? | Can an agent use the server to complete a realistic task? |
| Determinism | Controlled inputs should produce the expected result | The same task may produce different tool calls across trials |
| Result format | Pass or fail | Score or pass rate across repeated trials |
| Failure source | Handler code, schema implementation, transport, or authentication | Tool names, descriptions, schema usability, model, client, or agent instructions |
| Typical trigger | Commits that change server behavior | Changes to the model, prompt, schema, permissions, client, or server version |
MCP evals vs public MCP benchmarks
Public MCP benchmarks compare models or agent frameworks on a fixed set of tasks and shared servers. A common environment makes results comparable across models, but it does not cover your tool names, schemas, permissions, or user requests.
Salesforce AI Research's MCP-Universe spans 231 tasks across six domains and 11 real MCP servers, all scored by execution-based evaluators. In its published evaluation, the top-performing model finished 43.72% of tasks, so more than half went unsolved even under conditions the benchmark authors controlled.
| Comparison point | Public MCP benchmark | Application-specific MCP evals |
|---|---|---|
| Purpose | Compare models and agents under shared conditions | Determine whether a server configuration is ready to release |
| Tasks | Fixed and shared across participants | Based on user requests and observed failure cases |
| Servers | Third-party servers selected by the benchmark | Your server at a specific version and permission scope |
| Decision supported | Which model or agent to investigate further | Which server, model, or configuration change to release, revise, or roll back |
How MCP eval results get scored
Score the trajectory and resulting system state
Scoring only the final response is unreliable when tools change data or trigger external actions. An agent can report success after a failed call, and a retry can create an unintended second change.
Trajectory scoring examines the recorded decisions, tool calls, arguments, and errors. State assertions compare the resulting database record, file, calendar event, or other system state with the expected outcome. Together, they identify both the failed step and its effect.
Suppose an agent calls create_record and the request times out after the server writes the record. The agent retries, and the second call succeeds, but the final response reports one new record, while the database contains two. A response-only scorer may pass the final message, whereas trajectory and state scoring identify the duplicate write and fail the run.
Response-only scoring passes the final message, but the trace and database state expose a duplicate write.
Braintrust's agent evaluation guide covers additional success criteria for multi-step systems beyond MCP tool calls.
Combine deterministic checks, model-based scoring, and human review
MCP evals often combine three scoring methods because exact tool behavior and open-ended response quality require different evidence.
| Comparison point | Deterministic checks | Model-based judge | Human review |
|---|---|---|---|
| What it evaluates | Expected tool calls, arguments, call counts, and system state | Faithfulness to tool output and response quality | Release acceptability for ambiguous cases |
| Cost per run | Execution cost of the check | One model call per scored trace | Reviewer time |
| Variance | None when inputs and system state are fixed | Results may vary across repeated judgments | Decisions may vary across reviewers |
| Best use | Exact tool behavior and measurable outcomes | Open-ended responses with several valid paths | Judge calibration and disputed cases |
A deterministic check can reject a valid alternative route if it expects one exact call sequence. A model-based judge can evaluate multiple valid routes when its rubric defines the acceptable outcome, but you should validate it against human-labeled examples before its scores control a release. Human review establishes those reference examples and resolves cases where automated scores conflict.
One passing run does not establish reliable behavior
Sampling and context ordering can lead the same agent down different paths even when the task, server, model, and instructions remain fixed. One successful trial confirms that the agent completed the task once but provides no estimate of how often it will succeed.
Running each case several times produces a pass rate and exposes inconsistent behavior. A task that passes seven of ten trials has a measured success rate of 70%. Comparisons should use the same trial count, so score changes reflect agent behavior rather than a different sample size.
Repeated trials produce a measurable pass rate under a fixed task, server, model, and instruction set.
Braintrust trials support a global count and per-case overrides. Tools that write, delete, or spend money may justify more repetitions because an intermittent failure has a higher operational cost. Read-only tasks can use fewer trials after their results stabilize.
When to run MCP evals
Run MCP evals whenever a change alters the information or choices available to the agent. Handler tests can stay green through those changes, since the protocol keeps working correctly while the agent's behavior shifts underneath it.
- Model changes: A different model or model version can select tools, construct arguments, and recover from errors differently. Re-run the same task set before replacing the current model.
- Agent instruction changes: Small edits to system instructions can affect how readily the agent calls tools and how many attempts it makes. Compare the revised instructions against the existing experiment baseline.
- MCP client changes: Each client formats tool definitions, adds surrounding context, and returns server errors to the model differently. A client upgrade therefore needs evaluation results from the same server and tasks.
- Tool name or description changes: The model relies on names and descriptions to distinguish available operations. Test copy changes against the previous version to identify shifts in tool selection.
- Schema changes: New required fields, renamed parameters, and tighter enum or format rules can produce invalid calls. Add cases for the affected arguments before releasing the revised schema.
- Permission or scope changes: Reduced permissions introduce denials and new error paths. Evals should confirm that the agent handles the denied operation correctly and does not report a successful outcome.
- Server version changes: Adding or removing tools changes the choices presented to the model. Re-run the complete suite after changes to tool availability, discovery behavior, transport, or protocol support.
Braintrust can run MCP evals in CI/CD against an established experiment baseline. Use a focused suite on pull requests for fast feedback, then run the complete suite before release. Configure the release check to stop changes that fall below the approved scorer thresholds. For guidance on combining MCP evals with the other tests an agent needs, read our guide on how to test AI agents.
How to build an MCP evaluation framework with Braintrust
An MCP evaluation framework needs detailed traces, clear scoring criteria, comparable experiments, and release checks informed by production failures. Braintrust holds all four in one evaluation process, from the first server test through deployment.
Step 1. Instrument every MCP interaction
Use custom tracing to record tool calls, retrieval steps, and other agent logic as nested spans. Store the tool name and arguments as input, the server response as output, and relevant errors and metadata on the span. A span laid out that way ties a low score to the exact tool choice, argument, response, or failure behind it.
Step 2. Define scoring at the right level
Braintrust scorers can evaluate individual spans or an entire trace. Use Autoevals for supported quality checks, custom TypeScript or Python code for exact tool selection and schema assertions, and LLM-as-a-judge for criteria that require judgment. Apply the approved scoring criteria to experiments and production traces to keep release decisions and monitoring aligned.
Step 3. Test the MCP configuration in playgrounds
Use playgrounds to examine tool selection after a description or tool list changes. You can attach a public MCP server that supports OAuth to a prompt or project, then enable or disable exposed tools individually.
Braintrust's native MCP server connection is a UI-only feature available in playgrounds and UI-created experiments. For SDK-driven evaluations, connect to the server through your agent code and send the resulting traces to Braintrust.
Step 4. Compare release candidates under fixed conditions
Run each server version against the same dataset, scorers, and trial settings. Save each candidate run as an immutable experiment, then compare experiments to inspect task-level and aggregate score changes. Keeping the evaluation conditions fixed makes it easier to determine whether a server change improved performance or introduced a regression.
Step 5. Convert production failures into release checks
Filter production logs for failed or low-scoring MCP traces, then add confirmed cases to a versioned dataset. Run that dataset in CI/CD and configure the pass/fail policy as a merge requirement. Each confirmed production failure becomes a regression case that future server versions must pass.
Start tracing your first MCP agent run with Braintrust →
FAQs about MCP eval and MCP testing (2026)
What is an MCP eval?
An MCP eval measures whether an agent can complete a user task through a specific server often enough to meet a release threshold. The evaluation covers the agent's decisions and their effect on the connected system, so a convincing final response cannot hide an unsuccessful action.
How is MCP testing different from an MCP eval?
MCP testing validates the server contract, transport, authentication, schema handling, and responses to known requests. An MCP eval examines whether a model can interpret that contract and complete the intended task across variable runs. A server can pass its protocol tests even when an agent selects the wrong tool or changes the wrong data. The two also fail at different moments. Server tests break the moment handler code regresses, while eval scores can drop without a single line of server code changing, after a model upgrade or a reworded tool description.
What should an MCP eval measure first?
Prioritize the failure that would make an agent result unusable or unsafe. With Braintrust custom code scorers, teams can write TypeScript or Python checks that confirm the correct resource was selected for a read operation or that a write or delete operation used an authorized tool, passed valid arguments, and produced the expected system state before response quality is graded.
How many times should each task run in an MCP eval?
Five to ten trials per task is a useful starting range, and tasks with costly failure modes or unstable scores may require more. Keep the count fixed when comparing server versions. Budget accordingly, since ten trials across fifty tasks with a model-based judge means five hundred agent runs and five hundred judge calls per experiment.
Can MCP evals cover resources and prompts, or only tools?
MCP evals can cover all three primitives. Resource cases can verify URI selection, access permissions, content freshness, and whether retrieved information supports the answer. Prompt cases can test argument handling, output constraints, and behavior across the clients and models your application supports.
When should MCP evals run?
At minimum, run MCP evals before releasing a server change and after a production incident produces a new failure case. Teams with frequent changes can place a focused Braintrust evaluation in CI/CD on relevant pull requests, then reserve broader suites for release candidates or scheduled checks.