Skip to main content
Many of the advanced capabilities of Braintrust involve defining and calling custom code functions. Currently, Braintrust supports defining functions in JavaScript/TypeScript and Python, which you can use as custom scorers or callable tools. This guide serves as a reference for functions, how they work, and some security considerations when working with them.

​
Access functions

Several places in the UI, for example the custom scorer menu in the playground, allow you to define functions. You can also bundle them in your code and push them to Braintrust with braintrust push and braintrust eval --push. Technically speaking, functions are a generalization of prompts and code functions, so when you define a custom prompt, you are technically defining a “prompt function”.

​
Organizing functions into projects

Functions are organized into projects using the projects.create() method:
  • TypeScript
  • Python
import * as braintrust from "braintrust";

// Get a handle to the project (creates if it doesn't exist)
const project = braintrust.projects.create({ name: "my-project" });

// Use the project to create functions
project.tools.create({...});
project.prompts.create({...});
project.scorers.create({...});
If a project already exists, projects.create() returns a handle. There is no separate .get() method.
Every function supports a number of common features:
  • Well-defined parameters and return types
  • Streaming and non-streaming invocation
  • Automatic tracing and logging in Braintrust
  • Prompts can be loaded into your code in the OpenAI argument format
  • Prompts and code can be easily saved and uploaded from your codebase
See the API docs for more information on how to create and invoke functions.

​
Sandbox

Functions are executed in a secure sandbox environment. If you are self-hosting Braintrust, then you must: If you use our managed AWS stack, custom code runs in a quarantined-VPC in lambda functions which are sandboxed and isolated from your other AWS resources. If you run via Docker, then the code runs in a sandbox but not a virtual machine, so it is your responsibility to ensure that malicious code is not uploaded to Braintrust. For more information on the security architecture underlying code execution, please reach out to us.