Many of the advanced capabilities of Braintrust involve defining and calling custom code functions. Currently,
Braintrust supports defining functions in JavaScript/TypeScript and Python, which you can use as custom scorers
or callable tools.
This guide serves as a reference for functions, how they work, and some security considerations when working with them.
Access functions
Several places in the UI, for example the custom scorer menu in the playground, allow you to define functions. You can also
bundle them in your code and push them to Braintrust with braintrust push and braintrust eval --push. Technically speaking,
functions are a generalization of prompts and code functions, so when you define a custom prompt, you are technically defining
a “prompt function”.
Organizing functions into projects
Functions are organized into projects using the projects.create() method:
import * as braintrust from "braintrust";
// Get a handle to the project (creates if it doesn't exist)
const project = braintrust.projects.create({ name: "my-project" });
// Use the project to create functions
project.tools.create({...});
project.prompts.create({...});
project.scorers.create({...});
import braintrust
# Get a handle to the project (creates if it doesn't exist)
project = braintrust.projects.create(name="my-project")
# Use the project to create functions
project.tools.create(...)
project.prompts.create(...)
project.scorers.create(...)
If a project already exists, projects.create() returns a handle. There is no separate .get() method.
- Well-defined parameters and return types
- Streaming and non-streaming invocation
- Automatic tracing and logging in Braintrust
- Prompts can be loaded into your code in the OpenAI argument format
- Prompts and code can be easily saved and uploaded from your codebase
See the API docs for more information on how to create and invoke functions.
Sandbox
Functions are executed in a secure sandbox environment. If you are self-hosting Braintrust, refer to the self-hosting guide for information on configuring the sandbox environment for your deployment.
Custom code runs in quarantined environments that are sandboxed and isolated from your other infrastructure. For AWS deployments, this uses Lambda functions in a quarantined VPC.
For more information on the security architecture underlying code execution, please reach out to us. 
