How to test multi-agent systems

Why multi-agent systems fail after individual agents pass

Consider a customer support system with four agents. A triage agent routes the request, an order agent retrieves account details, a billing agent handles refunds, and a response agent drafts the customer reply. They coordinate through a shared case record containing the information and actions collected during the run.

An isolated evaluation can confirm that each agent selects the correct tools and returns valid output. Because these tests examine one component at a time, they do not expose what happens to context, shared state, or responsibility as work moves between agents.

Incomplete handoffs: The triage agent classifies the request correctly but omits the order number from its handoff, forcing the order agent to request information the system already received.

Shared-state conflicts: One agent overwrites information another agent recorded, leaving downstream agents with an incomplete or outdated case record.

Repeated work: The billing agent retrieves order details the order agent already collected, adding unnecessary model calls and latency.

Coordination loops: Two agents repeatedly transfer the task because neither recognizes that it owns the next action.

Incorrect final outcomes: Every individual step appears reasonable, but the response agent confirms a refund that the billing agent never completed.

The guide to testing AI agents explains how to evaluate tool selection, arguments, and complete trajectories within an agent. Multi-agent testing adds the coordination layer, with checks that show whether information, state, and responsibility move correctly across the whole system.

Designing representative multi-agent test tasks

A representative test case gives the complete system a job that requires coordination between agents. For example, the system may need to refund order 4127 for late delivery and send the customer a confirmed resolution. A task that one agent can complete independently belongs in that agent's component suite.

Define the request, initial state, available tools, and expected final state for every case. These constraints make the test repeatable and separate coordination changes from test-environment noise.

Control tools and starting state

Braintrust's guide for evaluating agents recommends controlling external dependencies so you can compare behavior across runs. Stub or sandbox databases, APIs, message queues, and services that could issue refunds or contact customers, using enough representative state to support the task.

Shared resources require the same control. The case record, memory store, or message queue should begin from a known state and reset between trials so no run inherits data or actions from the previous one. This also prevents evaluations from creating real-world side effects.

Define a verifiable final state

Successful completion must be visible in the tools and shared state. For the refund case, the refund record should contain the correct amount, order status should show the completed refund, and the customer reply should include the confirmation number. The shared case record also needs validation because a polished response can hide a failed or skipped action.

Turn each requirement into a separate scorer so the result identifies what failed. Deterministic scorers can verify state changes, schemas, and required actions, with model-based scorers assessing open-ended qualities such as the customer reply. Braintrust records each score independently, keeping task completion, shared-state integrity, and response quality visible throughout the evaluation.

Agent-level, handoff-level, and system-level testing

Multi-agent evaluation works at three levels. System tests show whether the user's goal was achieved, and agent and handoff tests locate the decision or payload that caused a failure. Together, they provide both a release signal and a useful diagnosis.

Agent-level tests

Hold the surrounding workflow constant and evaluate one agent against the inputs it actually receives. Score its tool selection, argument construction, output format, and state updates. Use payloads captured from upstream agents because hand-written inputs often exclude the incomplete or unexpected context that causes failures in real runs.

Store representative payloads in Braintrust datasets and reuse them as prompts, models, tools, or orchestration logic change.

Handoff contract tests

Define each handoff as a contract containing the required fields, accepted types, and context the receiving agent needs to continue. Deterministic checks can validate the payload before the receiving agent runs, including cases with missing fields, malformed values, or information unsupported by the original request.

These tests execute quickly and make structural failures easy to attribute. A missing order number points directly to the sending agent's payload construction and is caught before it can surface later as an unexplained system failure.

System outcome tests

Run the full task from its controlled starting state and verify the resulting tool actions, shared records, final response, and termination reason. The system-level score determines whether the agents completed the user's request correctly.

When a system-level test fails, the agent and handoff scores show where the run first diverged from the expected behavior. Passing all three levels gives the team evidence that each component worked, the handoffs preserved the required context, and their combined actions produced the correct outcome.

Scoring dimensions for multi-agent coordination

Separate scores turn a failed system outcome into a useful diagnosis. Braintrust scorers and classifiers can evaluate individual spans or complete traces using built-in scorers, model-based criteria, or custom code. Multi-agent systems typically require six coordination dimensions.

Tool-use and shared-state scorers require the intermediate actions recorded during the run. Add each agent's tool calls and state writes to evaluation metadata or trace spans so the scoring functions can inspect them directly.

async def task_func(input: str, hooks=None) -> str:
    # ...
    if rsp.choices[0].finish_reason == "tool_calls":
        tool_calls = rsp.choices[0].message.tool_calls
        hooks.metadata["tool_calls"] = tool_calls
    # ...

End-to-end completion remains the release signal, and the other dimensions show where coordination failed. This separation also reveals runs that reach the expected outcome through redundant, inefficient, or unsafe behavior.

Connected traces for failure attribution

An end-to-end score confirms that the system failed, but diagnosing the failure requires finding the first decision that sent the run off course. Separate logs from each agent force engineers to reconstruct that sequence by matching timestamps and identifiers. Capturing the complete task as one trace places routing decisions, handoffs, tool calls, and state updates in execution order.

A connected trace places each agent's model calls, tool calls, and costs under one root span for the user's task.

Create a root span for the user's task and use application-level tracing to capture each agent's activity in child spans. The spans should identify the acting agent and record its input, output, tool calls, state changes, and errors. When the final response confirms a refund that never occurred, the trace shows whether the billing agent skipped the action, received an incomplete payload, or acted on outdated shared state.

Agents running in separate services or processes must propagate the trace context across each handoff. Braintrust's advanced tracing documentation explains how one service can export the active span, and another can resume the trace as its child, preserving the complete execution record across service boundaries.

import requests
from braintrust import current_span, init_logger, start_span, traced

logger = init_logger(project="my-project")


# Client: Export the span
@traced
def process_request(request):
    return requests.post(
        "/api/process",
        json=request,
        headers={"X-Trace-ID": current_span().export()},
    )


# Server: Resume the trace
def handle_request(req):
    trace_id = req.headers.get("X-Trace-ID")

    with start_span(parent=trace_id) as span:
        result = process_data(req.body)
        span.log(input=req.body, output=result)
        return result

With the trace hierarchy in place, reviewers can start with the failed outcome and trace back to the first incorrect route, payload, tool call, or state update. The same intermediate data also supports agent-level and handoff scorers, connecting the final score to the behavior that produced it.

Evaluating multiple valid trajectories

A multi-agent system can reach the correct final state through more than one sequence. Independent agents may complete their work in different orders, and response agents may express the same result differently. Scoring every run against one fixed trajectory would incorrectly fail valid behavior.

Define the expected trajectory as a set of constraints. Required dependencies keep their order, such as completing the refund-policy check before issuing the refund. Independent actions only need to appear in the trace, regardless of which one runs first. A custom trace scorer can inspect the recorded spans and verify these requirements without enforcing one exact path.

The scorer should still reject patterns that indicate broken coordination:

• Missing steps: A required action never occurs, such as issuing a refund without checking the applicable policy.
• Redundant work: Multiple agents repeat the same lookup or state update without contributing new information.
• Unsafe actions: An agent uses a tool outside its permissions or completes an action that required escalation.
• Coordination loops: The same agents exchange the task repeatedly without changing the state or adding useful context.

Set an expected step-count range for each case and compare it across experiments. A sustained increase can reveal unnecessary routing, repeated work, or an emerging loop before the task-completion score begins to fall.

Repeated trials and coordination consistency

A single passing run says little about coordination reliability. A different routing decision can change the handoff order, tool calls, shared-state updates, and final outcome. Repeating the same case from an identical starting state shows how consistently the agents complete the task.

Braintrust's advanced evaluation process supports repeated trials globally or for individual test cases through trialCount in TypeScript and trial_count in Python. Lower trial counts keep pull request checks fast, with higher counts reserved for full post-merge or scheduled runs. The example below runs each input three times.

Eval("Chat assistant", {
  experimentName: "gpt-4o assistant - no history",
  data: () => experimentData,
  task: runTask,
  scores: [Factuality],
  trialCount: 3,
  metadata: {
    model: "gpt-4o",
    prompt: "You are a helpful and polite assistant who knows about sports.",
  },
});

Review the results as a distribution for each case. Occasional failures suggest intermittent routing or model behavior that requires comparison across traces. Repeated failures at the same step point more strongly to an orchestration defect, such as an incorrect routing rule, incomplete handoff schema, or missing termination condition.

Track pass rate and score variation alongside the average result. A stable overall score can hide one case whose behavior changes significantly across trials, making per-case consistency the release signal to watch in coordination-heavy systems.

Comparing prompt, model, tool, and orchestration changes

Coordination can shift when any part of the multi-agent system changes, even when the user-facing task stays the same.

Run each candidate against the same version of a Braintrust dataset so the inputs, starting states, and expected outcomes remain consistent. Record the model, prompt version, tool schema, and orchestration revision in the experiment metadata to preserve the configuration behind every result.

Change one variable per experiment whenever possible. Testing a model swap and prompt revision together makes it hard to identify which change caused a regression. Braintrust's experiment comparison shows score movements and output differences case by case against the selected baseline, making routing, handoff, and completion changes easier to attribute.

Experiment comparison grades each candidate against the baseline and shows which metrics moved.

When several changes must ship together, test them individually first and evaluate the combined configuration as the final release candidate. This keeps the investigation path clear if the combined configuration regresses.

Pre-release evaluation and production regression cases

A multi-agent evaluation suite stays effective when CI enforces known requirements and production monitoring supplies cases the dataset does not yet cover.

Gate releases with the evaluation suite

Run a high-signal subset on pull requests, then run the full suite after merge and before release. The Braintrust CI integration posts score changes against the baseline as a pull request comment. By default, bt eval returns a non-zero exit code only when an eval throws an exception, so blocking a merge on a quality threshold requires a custom Reporter() whose reportRun returns false when results fall below the release criteria. With that reporter in place and the check required by repository branch protection, a drop in task completion, safety, or termination can prevent the merge. Step count, latency, and cost can remain advisory until the application has fixed limits.

Score production behavior

Online scoring records each scorer run as a score span inside the production trace.

After release, online scoring applies scorers to sampled production traces asynchronously, without adding latency to user requests. Configure log alerts for low scores, errors, unusual costs, or other conditions that require investigation. User feedback can attach ratings and comments to the trace that produced the response, giving reviewers the context behind a reported failure.

Convert reviewed failures into regression cases

Review each production trace before adding it to the evaluation dataset. Once the failure and expected outcome are confirmed, record the responsible agent or handoff, the correct final state, and the related incident or support ticket in the test case.

Add the case alongside the fix so the next evaluation verifies the failure is resolved. Future changes then run against the same case, turning a production incident into a permanent check against recurrence.

Testing multi-agent systems with Braintrust

Braintrust turns each confirmed coordination failure into a reusable release check through a three-part cycle:

1. Locate the failure: Open the failed experiment, compare it with the accepted baseline, and follow the connected trace to the first routing decision, handoff, tool call, or state update that changed the outcome. This separates the original coordination failure from its downstream effects.

2. Confirm the expected behavior: Review the agent-level, handoff-level, and system-level results to define the correct route, required payload fields, permitted actions, and final state. Update the relevant scorer or threshold so future runs are measured against that requirement.

3. Prevent recurrence: Add the reviewed trace to the dataset with the corrected expectation and incident metadata. Run the case in CI against future prompt, model, tool, and orchestration changes, then apply the relevant scorer in production to detect the same behavior if it returns.

Notion uses Braintrust to align 70 engineers on evaluation and deploy frontier models within hours of release. Teams at Stripe, Vercel, Instacart, Zapier, and Ramp evaluate production AI applications on Braintrust.

Teams can start testing multi-agent coordination with Braintrust on the free Starter plan, which includes 1 GB of processed data and 10,000 scores per month, along with unlimited users. Start free with Braintrust →

FAQs: How to test multi-agent systems (2026)

How is testing a multi-agent system different from testing a single agent?

Single-agent testing evaluates decisions inside one bounded execution with controlled inputs and tools. Multi-agent testing adds two things a component suite cannot produce: evidence that each handoff carried the context the next agent needed, and evidence that the shared record still reflects what actually happened. Agent pass rates do not aggregate into a system pass rate, so a suite where every agent scores well can still ship a system that never finishes the task.

How do you test handoffs between AI agents?

Define both the structure and meaning of each handoff. Schema checks should validate required fields and types, with provenance checks confirming that values came from the user request, a tool result, or approved shared state. Test the receiving agent with incomplete, malformed, duplicated, and delayed payloads to confirm that it requests clarification, rejects invalid context, and avoids repeating completed actions. Human review can establish whether sampled handoffs contain enough context before that judgment becomes an automated scorer.

How many trials should each multi-agent test case run?

Set the trial count from the lowest failure rate the team needs to detect. For example, observing a behavior that occurs in 10% of runs with approximately 95% confidence requires about 29 trials. That volume may be impractical for every pull request, so smoke tests can use fewer trials and reserve larger samples for release-critical or scheduled evaluations.

How do you evaluate a system where agents can take different valid paths?

Build the constraint set from traces that have already passed review, recording which actions must appear, which pairs have a fixed causal order, which actions are never permitted, and what the final state must contain. When a run reaches the correct outcome through an unfamiliar route, review it once and either accept it as a valid path or add a scorer that rejects it. Score duplicate work, permission violations, coordination loops, and excessive step counts separately, so a flexible path still must meet efficiency and safety requirements.

When should the multi-agent evaluation suite run?

Run affected agent and handoff tests on every relevant pull request, then execute the complete suite for changes to routing logic, agent roles, shared-state schemas, tool permissions, model aliases, or termination rules. Scheduled evaluations help detect provider-side behavior changes that occur without a repository update. A production incident should also trigger an immediate replay of the failed case before approving the fix.