How Cloudflare evals its flagship agent
With Kylie Czajkowski, Engineering Manager, Agent Experience
Cloudflare runs one of the largest networks on the internet, with a developer platform used by millions. Kylie Czajkowski is on the Agent Experience team, working with the same primitives customers use to build agentic experiences.
The flagship is the dashboard agent. Cloudflare's feature surface is enormous, and the agent's job is to help developers access the right tool for their needs. That could be configuring a new service, deploying a worker for the first time, connecting bindings to storage, or debugging a production instance through the platform's own observability tools.
Since launch, it has gone from a discoverability aid to an agent that invokes four different MCP suites to take actions directly. Evaling an agent with that much surface area is hard. That's where Braintrust comes in.
From beta-testing to measuring resolution
Cloudflare's first read on agent quality came from a beta program. Product experts tried features in their own area and reported whether answers seemed right. But the limitations of this approach quickly became apparent.
It was nice for fine-tuning the overall vibe, but it wasn't actually giving us deterministic answers about whether the agent is solving problems. That was something we leaned on Braintrust for.
The team now runs an LLM-as-a-judge over entire conversations to produce an overall resolution score, so they can measure if a user got what they came for, whether that's a generation, an answer, or just leaving satisfied. Contrasting resolution against satisfaction tells the team if the agent is genuinely solving problems, or merely providing a pleasant interaction.
With Braintrust, the team can go a level deeper and look at resolution by product area. If a user asks about a DNS registrar and a specific set of rules, Cloudflare deterministically confirms that the rule the agent instrumented was correct and true to what the platform supports. The same layered approach covers the knowledge itself. They make sure that everything publicly documented is available to the agent, and that the agent captures the fine-tuned platform nuance that traditionally required an account executive or a customer success engineer.
Before Braintrust
Subjective, no gates
Iterating directly in development
The dashboard agent's behavior lives in skills, recipes, and prompts, and every change runs through evals in CI/CD. Cloudflare built a shared component, now used across platform internals, that measures skill performance before and after a change.
It's smart about both cost and quality. The diff only runs when a change touches a skill, prompt, or recipe, so the team isn't burning tokens on every commit. When a tweak meaningfully regresses responses, the change is blocked.
Instead of shipping changes, measuring them in production, and going back to the drawing board, we get to iterate directly in development, see how it performs, and then ship it to production.
The discipline has forced the team to write test cases for what Kylie calls "the corner of a corner case." It's these kinds of edge cases where Braintrust revealed the agent was circumventing good logic, allowing the team to achieve a higher level of performance.
A benchmark suite for models and sub-agents
Inside the team's admin portal, the same eval suite doubles as a benchmark for tracking conversation trends, tool invocation, and aggregate performance across product categories. As Cloudflare explores sub-agents with narrow tool sets and focused outputs, the team measures each prompt and sub-agent against models of varying capacity before choosing what powers it.
As Kylie put it, "We could see time to first token and response latency before. But looking at the quality of response, and making sure users are getting the resolution they're looking for, that's something we only unlocked when we implemented evals in this benchmark suite."
One agent, every product team
Because the dashboard agent touches everything, its conversations are a wealth of knowledge for every product team at Cloudflare. Kylie recently stood up an internal MCP server so product teams can query what's happening in their own slice of the platform, and identify where users get stuck, where tools don't match real needs, and what part of an API is effectively inaccessible to the agent and needs more context to become agent-friendly.
The same eval muscle powers Cloudflare's internal AI. Cloudflare OS automates workflows that used to be manual runbooks, with Braintrust confirming each automated section lands where the runbook says it should. And the internal AI code reviewer gets the same treatment. Evals find its weaknesses, and the team tunes the skill based on the Braintrust output.
Key takeaways
- Measure resolution, not just satisfaction. An LLM-judge over an entire conversation tells you whether users' problems were solved.
- Gate changes in CI/CD. Run the eval diff only when behavior-defining files change, and block regressions before they reach production.
- Benchmark sub-agents against models of varying capacity. Narrow agents with focused tools are easier to eval, and easier to power with the right-sized model.
- Share the knowledge with every team. Conversations with a platform-wide agent reveal where each product can improve, for humans and agents alike.
Thank you to Kylie for sharing Cloudflare's story.
Eval agents across your entire platform
Learn how Braintrust helps teams measure resolution across a platform-wide agent, gate every skill and prompt change in CI/CD, and benchmark sub-agents against models of varying capacity before they ship.
Read more customer stories
“Every single AI prompt we check into the codebase needs a Braintrust playground ID. If it doesn't, the test fails.”
How Dropbox built an evaluation pipeline for AI search
“Eval-driven development is the new test-driven development. Any projects that we take up, the first step is identifying the eval set.”