Running large sets of evals can be expensive, especially when you’re using an LLM judge on each data point.
If you don’t need the results on a specific schedule, you can lower costs using provider batch APIs. OpenAI and Anthropic both offer batch processing at 50% below their standard API prices.
Workflow evals in Braintrust let you use those batch APIs for eval tasks and scorers. You submit the requests to the provider and collect the results when they are finished. Everything gets tracked in one Braintrust experiment. The API is now available for the TypeScript and JavaScript SDK and the Python SDK.
Large regression suites, model comparisons, and scoring jobs where you can wait for results are all great candidates for workflow evals. The provider applies the discount to requests sent through its batch API.
The trade-off is time. OpenAI, for example, gives you a 24-hour completion window, so batch results can take up to 24 hours to arrive. That means if you’re batching both response generation and your LLM judge, you have to wait for the responses before you can kick off the judge.
Workflow evals coordinate the batch API and your evals using two callbacks you provide: submit sends work to the provider, and collect retrieves the outputs or scores. The completion configuration tells the SDK whether to poll the provider or wait for a webhook to determine when results are ready.
Recent SDK releases added WorkflowBatchTask and WorkflowBatchScorer, which group multiple cases into each provider batch. These are available in TypeScript v3.37.0+ and Python v0.45.0+. You choose the batch size, and the SDK matches results to individual items by ID.
Once you’ve defined your eval and configured its store, you can start a run and check on it later:
const started = await evaluation.start();
const progress = await evaluation.poll({
runId: started.runId,
});
These code snippets show how you’d start and continue a run, but the full setup also includes your dataset, task, scorers, provider integration, and storage. Each poll() call checks on your pending work and processes anything that’s ready. You can run these on a schedule until the eval finishes. If your provider sends a webhook when a batch is ready, you can also use that to continue the run.
When you submit a batch request, you may want to close the process while the provider works. You can optionally add persistent storage, such as WorkflowEvalRedisStore, to save progress in Redis and resume the eval as batch results arrive. Another process can continue the run using the same store and eval definition.
The TypeScript and Python guides include complete examples for connecting a provider, saving progress, and collecting results through polling or webhooks.
Try workflow evals yourself, or contact the team to get help testing the API on your workload. If you need support for Go, Java, or Ruby, let us know.