How Eve builds reliable agents for lawyers

Eve is a legal AI company built for plaintiff lawyers. These firms work on contingency, so Eve's job is to help them reach the largest possible case value in the least amount of time.

Eve's voice AI answers the phone after hours, gathers client and accident details, and can sign a new client on the call. Eve's agent then analyzes matters that often contain tens of thousands of documents, building medical chronologies, summarizing eight-hour depositions, running legal research across case law and statutes, and drafting work product for the case.

Mauricio, an engineer who leads chat at Eve, describes the product as agents operating inside a matter, pulling from case management systems, documents, and trusted legal databases to produce work a lawyer can file. His job is making sure that work is good, and Braintrust is how he measures it.

From sampled outputs to a graded benchmark

In Eve's early days, the team judged quality the way most AI startups do. They read a few outputs, asked a lawyer to review them, and shipped to early customers.

That approach still has a place, but it does not scale to thousands of law firms, and it misses fine-grained errors. So Eve built Plaintiff Bench, a gold-standard eval dataset drawn from legal matters and expert-written criteria.

The benchmark is now the largest dataset of its kind, with over 400 plaintiff law tasks across 40+ states and a wide distribution across task types and failure modes. It covers instruction-following failures and retrieval failures like returning too little information. With this data, Mauricio and team can measure performance on the dimensions that matter to lawyers.

Eve also works with Anthropic, DeepMind, and OpenAI to evaluate models against the benchmark before release, and evals every agent change, so model selection rests on measured performance.

Before BraintrustAfter Braintrust

Before Braintrust

Handful of outputs sampled

Lawyer manually reviewed each

Fine-grained errors slipped through

No way to scale eval to thousands of firms

Doesn't scale, misses details

Before Braintrust

Handful of outputs sampled

Lawyer manually reviewed each

Fine-grained errors slipped through

No way to scale eval to thousands of firms

Doesn't scale, misses details

After Braintrust

Plaintiff Bench with 400+ expert-graded tasks

Tagged by failure mode and task type

Production mistakes become test cases

Coding agents run the iteration

Expert benchmark, agent-driven iteration

Measuring what matters in legal work

Every example in Plaintiff Bench is tagged by the capability it tests, so Eve can eval end-to-end while isolating individual failure modes. The benchmark covers six task types and eight failure modes, including:

• Citation quality and coverage: Every claim Eve makes links to where in the documents it came from. An answer might carry hundreds of citations, which are infeasible to check by hand but straightforward to score automatically.
• Instruction following: Lawyers ask multi-part questions, and every part needs an answer.
• Temporal reasoning: Cases span years, and models still make mistakes organizing events in time. Eve tests this with work product built to probe it.
• Retrieval: In a matter with tens of thousands of documents, did the agent find everything?

Eve also tracks length regressions, tool calls, and cost per output. A script turns experiments into a report comparing before and after, so improvements and regressions are visible in one place.

Picking models with evidence from experts

Because the metrics are in place, evaling a new model is routine. When new models are released, the team runs their agent across all hyperparameters on Plaintiff Bench to see how each model performs.

Recent results showed that newer models scored higher on citation coverage and followed Eve's instructions around legal claims more faithfully.

That gave the team confidence that swapping in the latest models would mean better results for customers. The same process has shown frontier models steadily improving on long-context work like summarizing an eight-hour transcript, whereas older models degraded as context grew.

Turning production failures into test cases

Eve treats LLM quality issues as future test cases. Because Braintrust automatically preserves all the context used to generate a given output, failures are reproducible by default.

This allows their team to rerun experiments to confirm if fixes are working, and run the full Plaintiff Bench dataset to check for regressions.

A/B testing and online scores round out the measurement. Eve runs production scorers that detect frustration, apologetic responses, sentiment, and criticism from lawyers. This gives the team a continuous read on how agents perform in production, and evidence that they have been getting better over time.

Agents improving agents

Mauricio's favorite workflow pairs the Braintrust CLI with coding agents like Claude Code and Codex. First, his team iterates on metrics until they reliably capture the scope and quality of a legal task. Once a metric like citation coverage has been scoped, he can hand the whole improvement process to an agent.

At any given time he has around five agents running in parallel, reviewing their work as pull requests.

Proactive case auditing

Eve's newest product, Auditor, reverses the prompting model. Instead of waiting for a lawyer to ask a question in Eve, Auditor surveys every case overnight, looking for drivers of case value that may have gone undetected.

Firms used to hire human auditors for this. Now every case gets checked, and Eve has surfaced serious undiagnosed injuries that led to better treatment for clients and bigger settlements.

Key takeaways

• Build your benchmark from expert feedback. Plaintiff Bench grows from lawyer review, so the eval set reflects what lawyers need.
• Tag examples by failure mode and task type. End-to-end scores tell you whether quality moved. Tagged subsets for citations, temporal reasoning, and retrieval tell you why.
• Turn every production mistake into a test. Reproduce it, fix it, rerun the experiment. Never ship the same bug twice.
• Hand the iteration to agents. With trustworthy metrics in place, the Braintrust CLI plus a coding agent can hypothesize, implement, and measure for hours unattended.

Thank you to Mauricio for sharing Eve's story.