Independent eval research. Open methodology, published datasets, and the tools to run your own studies.
Original, open-source studies
Model comparisons, agent setups, cost, and modalities, measured on real tasks. Read the methodology and results, or rerun a study on your own data.
A coding agent rebuilt 27 complex designs through each MCP server, three trials each. Visual similarity scores CLIP embedding distance to the reference screenshot, and faithfulness comes from a claude-sonnet-5 judge.
0–1 CLIP score · 0.93 practical ceiling
0–1 · complex designs
225 procedurally generated tasks span arithmetic, symbolic rules, and data transforms, graded by exact match in code with no partial credit. Each model runs every task three times.
225 code-graded tasks · higher is better
Six providers transcribed 240 clips across eight content domains. Answer equivalence checks whether a downstream agent reaches the same answer from the transcript as from the ground-truth text.
Downstream answer correctness · 0–1
Applying the best-performing Parallel configuration from the World Cup eval to inspect the USA vs Belgium matchup as a source-backed research map.
We used Braintrust to evaluate six Parallel web research configurations across 48 World Cup matchups
A Braintrust-native eval comparing GLM-5.2, Opus 4.8, and Sonnet 5 on exact long-context retrieval, cost, and latency.
The full methodology, hypotheses, and findings for Braintrust's GLM-5.2, Opus 4.8, (and Sonnet 5!) long-context retrieval eval.
Call GLM-5.2 directly in playgrounds, prompts, and scorers, with no configuration, through July 31.
We pulled 1,781 real agent traces from Hugging Face into Braintrust, scored every run, and found that the wrapper around your model explains 7× more variation in success than the model itself.
Videos
Which GPT-5.6 model should you actually use? We ran an independent eval of Sol, Terra, and Luna against Anthropic's Fable, Opus 4.8, and Sonnet 5 on 225 machine-checkable tasks across arithmetic, symbolic rules, and data transforms, each at three difficulty tiers. We cover the dataset, the scoring, the decision map for the cheapest model that clears a 90% reliability bar, and what Fable's refusals do to the headline scores.
Which design MCP server builds a better front-end page: Figma or Paper? We ran an independent eval in Braintrust with a headless Claude Code agent rebuilding 40 simple and 27 design-heavy web pages through each tool, then scored every rebuild three ways: CLIP visual similarity (turning each image into a list of numbers so look-alikes score close), an LLM faithfulness judge, and two deterministic checks. We cover the dataset design, the scoring, the 0.93 similarity ceiling, what the traces revealed, and where the two land on consistency, speed, cost, and code quality.
Do open-source models fall apart once the context gets long? We put GLM-5.2 (the open-source model from z.ai) head-to-head against Claude Opus 4.8 on exact retrieval from long code contexts to find out. In this video, we build the whole eval in Braintrust — mechanically generating machine-checkable questions straight from CPython's abstract syntax tree, scoring answers three different ways, and adding a perturbation control to rule out memorization. We cover dataset design, scoring, statistical honesty (Wilson intervals), and the cost vs. accuracy trade-off.
Run evals like us
Each skill packages best practices from leading AI teams and researchers into a plain SKILL.md file your coding agent can read. Use them to get started running evals on your data.
Point your agent at a skill
Ask it to do what the skill describes, like validating a scorer against human labels or finding hidden failure modes.
- 1.Name the reference tier before computing anything.
- 2.Verify alignment: scorer outputs and reference labels must line up at the item and criterion level.
- 3.Report agreement (κ or α) with uncertainty, not raw accuracy.
- 4.Lead with the most decision-relevant false acceptance before any aggregate, and enumerate the dangerous cells case by case.
- 5.Break errors down by class and severity.
- 6.Test sensitivity and shortcuts: inject known regressions and improvements and confirm the scorer moves; probe whether length, confidence, or polish raise the score independent of quality; probe whether text addressed to the judge moves it; slice agreement by subgroup.