How Vercel keeps pace with AI's rapid model evolution

Vercel is the agentic infrastructure company. As the team behind Next.js, AI SDK, and v0, Vercel is the platform where humans and AI agents build, ship, and scale software together.

Malte Ubl, Vercel's CTO, and his team use Braintrust to manage new AI model releases with a robust eval infrastructure that can validate changes quickly, without slowing down the pace of product iteration.

The challenge: shipping every new model

One of the most frequent engineering tasks at Vercel is integrating new models as they're released. Customers expect to build with the latest models immediately, which means Malte's team needs to ship support for them fast, often within hours of release.

Without a strong eval foundation, each new model integration would mean starting from scratch. The team would be left manually spot-checking outputs, guessing at regressions, and hoping nothing slipped through. For a platform that thousands of developers depend on, that's not an option. Braintrust gives the team a structured way to benchmark new models against existing ones, catch performance differences early, and deploy with confidence.

Having a robust stack that can tell you whether your change is a good one is just fundamentally required to keep up with the pace of AI.

Embracing non-determinism

AI introduces a different approach to building software, in which the output of a product is non-deterministic. Though this can be challenging for engineers accustomed to traditional, deterministic systems, the power of AI and the things it makes possible are well worth the tradeoff. Agents are now capable of executing tasks autonomously that would have been impossible only two years ago.

It can be very uncomfortable because it's a new paradigm, but it does come with a lot of very obvious benefits. There's some discomfort with the nature of this technology that is not deterministic. But on the other hand, I can now do things that I couldn't do before.

Malte believes that building in this new paradigm means choosing tasks that are genuinely a good fit for AI, and embracing non-determinism. Good engineers now work with, rather than against, the potential for unexpected outcomes. AI coding tools, for example, work incredibly well, and when developers couple their non-deterministic output with a robust set of eval and observability practices, the productivity gains can be significant.

Evals support iteration velocity

For Vercel, evals serve as a core mechanism for maintaining speed, even as new models proliferate and new software becomes easier to build.

Evals enable the team to make changes and know whether those changes are improvements. A new model can be tested and then deployed with a full understanding of its strengths and weaknesses. And new code can be validated and shipped just as quickly as it can be written.

Ultimately the most important thing in software engineering is the velocity of my team. And if I'm making agents or AI software, then evals are at the fundamental layer of keeping up that iteration velocity.

Collaborating on the modern AI API stack

Vercel and Braintrust have an ongoing collaboration centered on the modern AI API landscape. As AI labs add features like tool calling, memory, and planning as first-class API capabilities, both Vercel's SDK and Braintrust's observability layer need to represent this increasingly rich data.

AI labs have been adding more and more features to their APIs. That makes it more difficult to represent on the client side. We're collaborating on how to represent the modern AI APIs in an efficient manner, both on the client side and the observer side.

Malte analogizes this to assembling a rocket in mid flight. Tool calling was a novel concept, and it wasn't obvious initially whether or not it would need deep observability. But based on feedback and joint experimentation, Braintrust and Vercel both built products to manage this new framework.

Observing customer behavior in production

Vercel's approach to quality is grounded in close observation of what customers are actually doing. The team listens to customer feedback, but also uses production traces in Braintrust to identify issues that might not reach the threshold of a direct complaint.

With evals and observability, teams no longer need to wait until customers identify a problem. They can see what's happening in real time, troubleshoot what's wrong, and ship fixes all before any user notices.

Key takeaways

Invest in eval infrastructure early. A robust eval stack is a prerequisite for keeping up with the rapid pace of model releases.
Work with non-determinism, not against it. Embrace the probabilistic nature of AI outputs and build processes that account for it.
Evals allow for iteration velocity. In AI engineering, the ability to validate changes quickly is the single biggest factor in team speed.
Look beyond direct feedback. Production traces reveal quality issues that customers may not report directly.

Thank you to Malte Ubl for sharing Vercel's story.