4 September 2026

Loop vs Braintrust MCP + Codex for production agent investigations

Izzy Hurley10 min

Key takeaways

Loop completed comparable investigations 39% faster

Production cost and release regression took 191 seconds in Loop and 314 seconds with MCP + Codex.

Loop matched MCP + Codex's insight score on 4 of 5 workflows

Loop passed 19 of 22 insight checks. MCP + Codex passed 20.

Loop linked 11 supporting traces

Each link opened the trace and its spans beside the analysis.

Loop starts inside a Braintrust project with product context and investigation skills already loaded. Braintrust MCP + Codex combines the same project data with Codex's repository, terminal, and connected tools.

I ran both setups through five production investigations to compare their conclusions, completion time, and ability to act on the results. Both used gpt-5.6-sol at the same reasoning effort. Deterministic checks graded each run against a handwritten answer key.

The study uses SWE-chat, a living dataset of real coding-agent sessions. It captures corrections, failure reports, interruptions, and other parts of development that polished benchmark tasks leave out. The paper reports that developers pushed back in 44% of turns, which made the dataset useful for testing investigations into regressions, failed handoffs, and production repairs.

Hypotheses

I expected Loop to finish analyses faster inside the project because it starts with product context and built-in investigation skills. Requests that cross into code or other systems should favor MCP + Codex. The same model and reasoning effort should produce similar conclusions from the same project data.

The five workflows covered:

Understanding cost, by finding the most expensive agent runs and whether models, tool retries, or long prompts drove them.
Catching release problems, by comparing agent behavior before and after a release for changes in quality, speed, or cost.
Learning how customers use the product, by finding unexpected workflows, tracking whether they are growing, and sending a weekly report.
Making sure work gets delivered, by finding runs where the agent finished the task and never returned the result, then testing and monitoring for it.
Fixing production failures, by tracing a failure back to the code, testing a fix against similar real cases, and checking that it improves results without costing more.

Each workflow ran once per setup and used a handwritten answer key with deterministic checks. Twenty-two read-only insight checks applied to both tools. I scored actions inside Braintrust separately from tasks that required a repository checkout. With five runs per setup, the results describe these cases and do not establish a general performance ranking.

Experiment design

Each workflow used the same paired setup.

Same task, two tools

Each workflow ran once in Loop and once in Braintrust MCP + Codex. Only the available context and tools changed.

Five production workflows

Identical 200-trace projects · gpt-5.6-sol · low reasoning effort

Same prompt

Loop starts inside Braintrust

Braintrust project context is already loaded
Built-in investigation skills select the query path
Citations open traces and Patterns beside the answer
Project tools can read and write Braintrust state

Braintrust MCP + Codex connects to the project

Braintrust MCP + Codex discovers and calls project tools
Braintrust MCP + Codex reads and writes the matched project
A repository and terminal are available when required

Check the answer and completed work

Used the required tracesReported the right facts and evidenceCompleted the requested work

Insight quality and executed actions are scored separately

Results

Workflow	Loop insights	Braintrust MCP + Codex insights	Result
Production cost	5 of 6	5 of 6	Loop reported the correct costs. Both linked inspectable evidence.
Release regression	5 of 6	5 of 6	Loop quantified the effect. MCP + Codex prioritized a change.
Emerging customer workflows	3 of 3	3 of 3	Both found and measured the same trends
Silent handoff failures	0 of 1	1 of 1	Braintrust MCP + Codex selected the correct confirmed-failure traces
Production repair	6 of 6	6 of 6	Both reached the same evidence-backed diagnosis

Loop completed the production cost and release regression investigations in 190.76 seconds, compared with 314.26 seconds for MCP + Codex. It finished those two comparable investigations 39.3% faster.
Loop passed 19 of 22 read-only insight checks, compared with 20 for MCP + Codex. The tools tied on four of five workflows. MCP + Codex passed one additional check in the silent handoff investigation.
Loop linked 11 supporting traces across the production cost and release regression investigations, including matched comparisons and counterexamples.
Both tools passed all three adoption checks. They identified the same two growing workflows and found that code-edit usage stayed flat.
MCP + Codex completed six of eight shared Braintrust actions, compared with two for Loop. It also passed eight repository checks that only the MCP setup could attempt because it received a checkout. Those repository checks measure the additional checkout capability separately from insight quality.

In Loop, selecting a citation opens the linked trace beside the answer. A Pattern connects a synthesized finding to its supporting evidence and a reusable view.

Loop and Braintrust MCP + Codex on the same task

Five workflows ran once in each tool with identical data and model settings. Select a workflow to compare the result.

Time

Loop finished 59% sooner

84.5s Loop · 204.1s Braintrust MCP + Codex

Insight checks

5/6 Loop · 5/6 Braintrust MCP + Codex

Read-access analysis, arithmetic, evidence, and calibration

Prompt both tools received

Look at this project's traces from the past 24 hours. Identify the most expensive runs and explain which model calls, tool retries, or context growth drove the cost.

Time to outcomedrawn to scale, 240s across

Loop84.5s

Ranked answer, costs correct

Open any linked trace to inspect its complete child-span tree.

Braintrust MCP + Codex204.1s

Ranking correct, cost arithmetic off

The child-span references are there to check, but the arithmetic has to be redone before you act on a number.

Where they agreed. Both found the frozen top expensive runs and avoided treating every long run as waste. They ranked the same trace first, counted its 36 model calls, and noted that it ended with “No response requested.”

LoopInsights 5/6

Canonical trace costs and linked evidence

Reported $34.2518 for the window, after noting that the $68.5035 span sum double-counts root and child rollups

Ranked the top three in the correct cost order, with a $4.1913 avoidable-context estimate on the leader

Used trace 767c87f9 as a counterexample. It cost $0.6002 despite four failed searches, so retries alone were not the driver

Linked every ranked trace. Selecting one opens its spans, input, and output beside the analysis

Braintrust MCP + CodexInsights 5/6

Exact window and child-span evidence

Reported $68.5035 for the window, the double-counted figure, while naming the doubling in its own arithmetic

Found the correct top three in order, with every per-trace cost reported at 2× the true figure

Ranked trace 767c87f9 sixth and attributed more than half its cost to four repository_search retries

Listed exact child-span IDs for every attributed driver

Insight checklist for this workflow

Check	Loop	Braintrust MCP + Codex
Used the exact 24-hour window	fail	pass
Queried all 200 traces	pass	pass
Found the top expensive runs	pass	pass
Used canonical cost arithmetic	pass	fail
Linked inspectable evidence	pass	pass
Did not treat length as waste	pass	pass

Where they diverged. Both made the evidence inspectable. Loop's native links open each trace beside the answer, including its child spans and raw input and output. Braintrust MCP + Codex specified the exact window and listed supporting child-span IDs, but its reported costs retained duplicated trace rollups.

Both used gpt-5.6-sol at low reasoning effort

Across 22 read-only insight checks available to both tools, Loop passed 19 and Braintrust MCP + Codex passed 20. Shared Braintrust actions and checkout-only execution are reported separately from insight quality.

How to choose

Start with Loop for production questions about cost, quality, release behavior, or customer usage. It starts with project context and built-in investigation skills. In this study, it completed the comparable analyses faster and kept cited evidence beside the answer.

Bring the investigation into MCP + Codex when the next step requires a repository, terminal, or connected system. In this study, that setup scheduled a report, created eval artifacts, and verified a production fix.

How the checks worked

I wrote a deterministic checklist for each workflow. The checks covered the traces queried, facts reported, evidence cited, and artifacts created. Code compared each result with a handwritten answer key without an LLM judge or human reviewer.

Twenty-two read-only insight checks applied to both setups. The production repair contract originally scored execution only, so I added six checks that graded both diagnoses against the same rubric. A citation passed when it included a direct trace or Pattern link or a specific trace or span identifier.

Loop passed 19 checks, and MCP + Codex passed 20. The pooled rates were 0.864 and 0.909.

Workflow	Insight checks	Loop	Braintrust MCP + Codex
Production cost	6	5	5
Release regression	6	5	5
Emerging customer workflows	3	3	3
Silent handoff failures	1	0	1
Production repair	6	6	6
Total	22	19	20

An equal-weight average of the five workflow insight rates is 0.733 for Loop and 0.933 for Braintrust MCP + Codex. The pooled rate is more stable because the handoff workflow contributes one insight check while three workflows contribute six each.

I scored execution separately. Eight checks covered actions available to both tools inside Braintrust. Loop passed two, and MCP + Codex passed six. Eight additional checks required the repository checkout available only to MCP + Codex, which passed all eight.

The production-cost checklist provides a concrete example. Its hidden answer key contained one record for each of the 200 traces in the requested window. Each record stored the true trace cost and the components needed to determine why it was expensive. Loop and Braintrust MCP + Codex could query the traces. Only the scoring program could read the answer key.

Scored fact	Hidden answer for the most expensive trace
Trace	`trace-005`
Total cost	$8.462853
Input-context cost	$7.268823
Output-generation cost	$1.194030
Cost attributed to retry calls	$5.988393
Extra cost versus a cheaper model	$4.638157
Model	`claude-4.5-sonnet`
LLM calls	36
Tool retries	6
Context growth	116,747 tokens
Dominant cost driver	Tool-retry followups

The key also stored the complete cost ranking. The correct top three were trace-005 at $8.4629, driven by retry followups, trace-007 at $6.8891, driven by model choice, and trace-009 at $6.4740, driven by context growth. Nine traces contained the relevant cost patterns. The other 191 made sure the tool had to search and rank all 200 traces.

Six checks compared each report with the hidden answer:

Check	Loop	Braintrust MCP + Codex
Used the exact 24-hour window	fail	pass
Queried all 200 requested traces	pass	pass
Found the top expensive runs	pass	pass
Used the canonical, non-duplicated cost arithmetic	pass	fail
Linked inspectable supporting evidence	pass	pass
Avoided treating every long run as waste	pass	pass

Loop and Braintrust MCP + Codex each passed five of the six checks. Loop linked the ranked traces and reported the correct cost. Its stated time window did not match the requested 24 hours. Braintrust MCP + Codex specified the exact window and named supporting child-span IDs. It reported $16.9257 for trace-005 after summing parent and child rollups, twice the canonical cost. Both conditions let a reader inspect the supporting evidence. Loop's links additionally open the trace and its child spans alongside the investigation.

The data

The SWE-chat dataset records complete, multi-turn interactions from developers who opted into public GitHub checkpoint logging. Each session can include user prompts, agent responses, tool calls, token usage, code diffs, and line-level human-versus-agent authorship. The SWE-chat paper describes the collection process and its findings about how developers steer, correct, and override coding agents in real repositories.

SWE-chat continually grows, so I pinned revision f66cca95b14caaa4177f7ed5eaa424608dadcffa for reproducibility. That snapshot contains 5,851 sessions across 201 repositories and roughly 2.69 million logged events from January through April 2026. It includes sessions from Claude Code, Codex, and OpenCode.

I used the dataset's behavior labels and structural fields to subselect sessions in two stages. The candidate pools included 24,334 pushback-labeled prompts across 4,095 sessions, 1,366 sessions linked to a resolved commit through an unambiguous single-session checkpoint, and 1,013 long trajectories. A trajectory entered the long-session pool if it met at least one 90th-percentile threshold: 150 tool calls, 8.56 hours, or 40 turns.

I then assembled five fixed projects of 200 top-level traces, rebalancing the mix of repositories, coding agents, and trajectory lengths. Depending on the workflow, 9 to 22 traces contained the facts needed for the answer. The rest were realistic distractors from the same corpus. Model-generated labels selected candidates only. Handwritten answer keys used fixed rules, counts, hashes, and expected artifacts, so no model-generated annotation became scoring truth.

Controls

Each tool ran in a fresh Braintrust project containing the same data.
Neither tool could read the hidden answer key or the other tool's project. The scoring program ran its tests after each tool finished.
Both tools used gpt-5.6-sol at low reasoning effort. Braintrust MCP + Codex also received a temporary workspace and a repository checkout.
A deterministic program compared each answer and completed action with a handwritten answer key.

Scope of the findings

Each case ran once using one pinned version of the dataset. The results describe these five runs and show what each tool produced. They do not estimate performance across every cost investigation, release analysis, or production repair.

The production-repair result compares the complete setups. Braintrust MCP + Codex had a repository and terminal, while Loop did not. It does not isolate the Braintrust MCP + Codex setup from the repository and terminal it received. Future versions of Loop, Braintrust MCP + Codex, and the underlying model can produce different results.

You can run the same production investigation in Loop or through Braintrust MCP + Codex. Sign up for free to compare them on your own agents, or book a demo to walk through your setup.