Eval foundations: How to eval agents for PMs
9 December 202610:00 AM PT
Noorain NooraniProduct Manager
Jess WangDeveloper Advocate
If you’re a PM building agents, your job probably looks a bit different than it used to. You still need to capture use cases for different personas, define acceptance criteria, and measure whether the agent meets them. But you can use evals to do that.
In this live workshop, Noorain and Jess will show you how to define quality criteria, build test cases and scorers in Braintrust, and compare results to understand how prompt or model changes impact your agent's performance.
What you'll learn
- How to translate product requirements into measurable quality criteria
- How to use Patterns and Loop to investigate where your agents fall short
- How to turn production failures and edge cases into an eval dataset
- How to calibrate automated scoring with SME feedback
- How to compare prompts and models to identify improvements and regressions
Recent on-demand workshops
Eval foundations: Answering questions about your agent
4 November 202610:00 AM PT
In this live workshop, Jess will share example evals developed at Braintrust for three different use cases. Testing an agent workflow, choosing a model judge, and comparing models.
Eval foundations
7 October 202610:00 AM PT
Designed for new Braintrust users, this workshop will cover the core foundations of evals and how to get started with Braintrust.
Put the workshop into practice
Start using Braintrust to trace what ships, score it against real data, and turn every regression into an eval you can rerun.