How Pylon iterates on every prompt with evals

Pylon is building the first agentic B2B support platform. AI for support used to mean chatbots that pull answers from a knowledge base. Agentic support requires using a company's backend tools to proactively solve customer problems.

That raises the stakes on quality. Pylon's customers use its support platform to serve their customers, so consistency in responses is the product. Fred Zhao, an engineer on Pylon's AI team, has watched the company adopt Braintrust as part of a disciplined and automated approach to evals.

The bug bash that changed the rules

In the early days, Pylon's approach to testing prompts was vibes-based. The turning point came during an engineering bug bash where the team made a concerted push to use Braintrust programmatically. The result was a new rule that requires using Braintrust to test every prompt as part of the CI pipeline.

A playground comes with a previously curated dataset, so when a future engineer iterates on a prompt, they rerun the same playground with their updated version and inherit all the prior work.

Fixing the rubber-banding problem

Before the team built this rigor, prompt development had a failure mode Fred calls rubber-banding. One engineer writes a prompt that achieves their goal but misses an edge case. A second engineer adjusts for that edge case, and over-pivots the other way.

For a support platform, that back-and-forth introduces inconsistency and elevates churn risk. Bringing Braintrust into the daily workflow has helped Pylon develop a quality bar that can be tested across the team.

Before BraintrustAfter Braintrust

Before Braintrust

Vibes-based prompt iteration

No shared eval baseline

Rubber-banding on edge cases

Inconsistency across engineers

Every prompt change risked regressions

Before Braintrust

Vibes-based prompt iteration

No shared eval baseline

Rubber-banding on edge cases

Inconsistency across engineers

Every prompt change risked regressions

After Braintrust

Playground ID required in CI

Curated datasets travel with each prompt

Failure modes captured as evals

Automated RCA and prompt sync skills

100% enforced, playgrounds by default

Datasets built from the 0.1%

Because Pylon runs Braintrust in production for observability, it has rich data on real-world usage patterns. That information is unusually diverse, since every customer supports their own customers differently. The team scripts against the Braintrust API to filter production traffic dynamically and surface the patterns that matter. Edge cases appearing in maybe 1% of traffic, or failure modes at 0.1%, get distilled into eval datasets that guard against regressions going forward.

The same approach applies to model selection. Pylon builds on all the leading AI labs, choosing model families and versions per use case based on quality and speed, with cost monitored.

Fred and team can use playgrounds to add another model or version, run it against the curated dataset, and see the results. For quality specifically, the team leans on LLM-as-a-judge scorers to bulk-check outputs, and uses Loop to spin up new scorers quickly.

The agent RCA skill

Pylon's engineers try to automate anything tedious, and on-call debugging was a prime target. The AI on-call team built a Claude Code skill called "agent RCA" that takes a bug report from Slack, finds the relevant identifiers, and calls a mix of logging MCPs plus the Braintrust MCP to reconstruct what happened.

Fred credits the consistency of Braintrust's spans and traces for making this possible. With homegrown event logging, the hardest problem is knowing what's worth logging, but Braintrust's standard means the timeline can always be reconstructed.

The CLI as an agent's tool

For Pylon, the Braintrust CLI is a powerful tool because agents can use it autonomously, allowing the team to codify repetitive workflows as skills.

Drilling into filtered production data to replicate a specific failure case, syncing a prompt to Braintrust mid-iteration, testing a temporary version, and syncing the final version back into the codebase are now all skills. And when they run a skill, everything happens automatically.

It's also shaping how Pylon thinks about its own product. The workflows Claude Code has made effortless for engineers are also what an agentic support platform should do for support teams. This points the way to future product development and the next iteration of Pylon's platform.

Key takeaways

• Make evals a merge requirement. A CI test with a Braintrust playground turns eval discipline from a habit into a rule.
• Playgrounds are institutional memory. Storing the playground ID with the prompt means every future engineer inherits the curated dataset and prior work.
• Filter production for edge cases. API filtering pulls 0.1% failure modes out of diverse production traffic. Datasets and scorers guard against new regressions.
• Hand your tooling to your agents. The Braintrust MCP powers automated root cause analysis and the CLI makes dataset curation and prompt syncing automatic.

Thank you to Fred for sharing Pylon's story.