Evaluate

Measure against quality baselines

Build datasets, define scorers, run experiments, and gate deployments. Measure quality and improve iteratively.

Scores and metrics	GPT-6 Sol Base	GPT-6 Luna Comparison
Comparison grade		Regression Comparison has better accuracy, factuality, helpfulness, instructionFollowing, tone, safety, completeness, citationQuality, latency, cost, errors, and load.
Accuracy	86%avg	92%-6% 3
Factuality	88%avg	93%-5% 2
Helpfulness	82%avg	89%-7% 2
Instruction following	80%avg	90%-10% 3
Tone	87%avg	90%-3% 11
Safety	94%avg	96%-2% 1
Completeness	79%avg	88%-9% 2
Citation quality	73%avg	85%-12% 2
Duration	2.4savg	1.8s-0.6s 2
Total cost	$0.031avg	$0.024-$0.007 2
Total tokens	1,840avg	1,420-420 2
Prompt tokens	820avg	690-130 1
LLM calls	1.5avg	1.25-0.25 1
Errors	0.5avg	0-0.5 1

The eval lifecycle

From “I think this is better” to “I know it is”

Build datasets

Golden sets from production logs, user feedback, or manual curation.

Create a dataset

Define scorers

Automated, LLM-as-judge, and human review in one place.

Write scorers

Run experiments

Compare prompts, models, strategies. Side-by-side diffs.

Run an experiment

Gate deployments

CI/CD quality gates block bad changes.

Set up CI gates

Measure quality

Three ways to score. Use them together.

Combine deterministic checks, LLM judges, and human review in one platform. The same scorers run offline in experiments and online in production.

95%Accuracy

62%Helpfulness

Instruction following

Automated scorers

Deterministic checks for format, structure, accuracy.

Build code scorers

Name

Helpfulness

Model

GPT 5 Mini

Score

62%

Rationale

The response directly answers the order status question with clear tracking details and a friendly follow-up offer.

LLM-as-judge

Score tone, helpfulness, reasoning quality.

Configure LLM judges

Response polish

Choose an option to rate the polish level of the response

Human review

SMEs evaluate and build labeled datasets.

Set up review

Compare everything, side by side

Prompts, models, strategies, quantified across every test case.

Experiments

Rich diffs

Run the same dataset through two configs and see what changed.

Prompt v3 compared to Prompt v4

Diff

Quality over time

Track regressions across releases

Every experiment is a data point. See the trend before you ship.

Tone87%

Helpfulness87%

Factuality88%

The complete eval toolkit

Everything you need to measure quality

Find a dataset

Production failures

User reported issues

High latency traces

Golden responses

Create new dataset

Datasets

Versioned test cases. Build from production logs or manual curation.

Explore datasets

% Accuracy

GPT 5.2

87.3%AVG

Claude 4.5 Opus

92.1%+4.8%

Gemini 3 Pro

85.0%-2.3%

Mistral Large

78.2%-9.1%

Experiments

Compare prompts, models, strategies. Side-by-side diffs.

Run experiments

Created

Input

Factuality

Helpfulness

Sentiment

Jan 25 2:34 PM

Hi! What's the status of order #12345?

94%

91%

82%

Jan 25 2:19 PM

My package arrived damaged

92%

84%

45%

Jan 25 2:08 PM

Where is my refund? It's been 2 weeks!

78%

55%

32%

Jan 25 2:31 PM

Can I return my order from last week?

91%

88%

75%

Jan 25 2:22 PM

Do you have any discounts for bulk orders?

89%

87%

88%

Scorers

Autoevals library. Same library shared offline and online.

Write scorers

Customer support bot

Diff

Base task

system

You are a helpful customer service assistant. Be friendly and concise.

user

Comparison task

system

user

Playgrounds

No-code prompt iteration. Tool calls + MCP.

Try playgrounds

How was the agent's response?

Human review

Structured annotation, review queues.

Set up review

braintrust-evalBotcommented1 hour ago• edited

Support agent (HEAD-1714341466)

Score	Average	Improvements	Regressions
Policy compliance	94% (+4%)	12 🟢	2 🔴
Resolution quality	87% (+6%)	18 🟢	5 🔴
Escalation accuracy	91% (+2%)	9 🟢	3 🔴
Duration	1.8s (-0.4s)	11 🟢	2 🔴