How Bubble turns user prompts into production apps

Bubble is an AI app builder that enables users to create full-stack web and mobile apps, deploy them, and run them at scale. Historically a no-code platform, Bubble now lets users build with prompts while maintaining a visual app-editing experience. Builders have turned their ideas into products that support businesses ranging from e-commerce to customer support agents.

A large part of what Bubble's Agent does is turn a user's prompt into elements in the Bubble app that bring that intent to life. That gets harder as scope and functionality grow. Bubble's longest-tenured users have very complex, large apps that transact millions of dollars, so maintaining accuracy is critical.

Zach Tessler and Ally Cody are engineers on Bubble's AI team who use Braintrust to measure their Agent's performance with evals. They also recently launched a new version of the Bubble Agent that significantly improved accuracy, without negatively impacting latency or increasing inference costs.

The limits of manual testing

Before Braintrust, the team at Bubble tested their product manually, iterating against a finite set of scenarios and unintentionally overfitting their Agent.

With Braintrust, they can create datasets that encompass more of the user experience. This helps avoid regressions as they make changes. It also gives them consistent benchmarks, so they can track how much progress they've made against their most important priorities.

Before BraintrustAfter Braintrust

Before Braintrust

Manual testing by engineers

Testers from across the company

Same go-to prompts retested

Agent overfit to few cases

Narrow coverage, slow progress

Before Braintrust

Manual testing by engineers

Testers from across the company

Same go-to prompts retested

Agent overfit to few cases

Narrow coverage, slow progress

After Braintrust

Datasets across the user base

Scorers for equivalent outputs

Quality, latency, and cost tracked

Nightly evals and auto-triage

1.5x more accurate Agent after evals

Defining correctness when there's more than one right answer

Bubble defines all of its metrics as scorers against eval datasets in Braintrust. The team defines test datasets that capture the behaviors they need to support, then scores the visual quality of the outputs and the functionality.

Ultimately, the resulting app needs to match what the user wants to build. The team also tracks latency and inference costs in Braintrust alongside these quality metrics, to help manage the tradeoffs between cost, speed, and outputs.

Because there are often several valid ways to build the same thing, the evals have to be robust to equivalent outputs that might look a little bit different.

All of these scores are defined in code, and the team uses Braintrust to compare experiments. They can change part of the codebase, swap in a new model, or develop a new feature and see how that affects the aspects of the Agent's performance they care about. When an improvement emerges that may lead to tradeoffs between quality and latency, they can make a judgment about the best experience for their users in a quantifiable way.

Inspecting traces and conducting automated triage

Traces in Braintrust let the Bubble team piece together exactly what the Agent did, where it may have made a mistake, where it had to backtrack, and where the Agent's behavior was inefficient. They can also see latency and accuracy averaged over many scenarios, which helps them decide which model to use.

The team runs two automations. One scans live traces for errors, and when it finds a pattern it puts up a PR for the team to review. The other runs nightly evals, allowing coding agents to dig into the trace views and automatically open tickets for bugs they find.

The product team at Bubble also uses Braintrust. They assess user sentiment and understand the most common types of requests coming in, which helps them prioritize where to focus and what to build next.

Shipping Agent 2

Bubble recently launched Agent 2, a completely new version of its core app-building Agent. It's 1.5 times more accurate than its predecessor at visually representing user prompts, while also performing faster and more cost efficiently. With Agent 2's new functionality, users can do almost anything with the Agent that they used to do manually in Bubble's visual editor.

As new models were released during the product development process, the team could quickly eval whether they would improve the quality of the Agent's outputs, and used Braintrust to measure the related latency and costs.

Those experiments ran on both fixed test cases and live user requests. Doing both allowed them to make a large number of changes in the final weeks of the launch timeline while ensuring the Agent was improving with each update.

Next, Zach and Ally are continuing to focus on driving quality up and latency down, and building new entry points for users to interact with the Agent. Version control is also on the roadmap, which they expect will add a layer of complexity that requires leaning into their evals even more.

Key takeaways

• Build evals around user intent. Define the request and the exact thing it should produce in your platform, then measure whether the agent produced an accurate result.
• Score for equivalence rather than exact match. For Bubble's users, there are often several valid ways to build the same thing. They needed to build scorers that could account for this variance.
• Build datasets that cover your whole user base. Manual testing on limited data is inefficient and can overfit to repeated prompts.
• Make trade-offs quantifiable with evals. A model-swap can introduce unexpected tradeoffs. Evals help turn these intuitions into something Bubble can measure.
• Use coding agents to automate triage. Bubble uses the Braintrust CLI to scan a day of production traces for failure patterns, and opens a PR every night with fixes.

Thank you to Zach and Ally for sharing Bubble's story.