Functions
Many of the advanced capabilities of Braintrust involve defining and calling custom code functions. Currently, Braintrust supports defining functions in Javascript/Typescript and Python, which you can use as custom scorers or callable tools.
This guide serves as a reference for functions, how they work, and some security considerations when working with them.
Accessing functions
Several places in the UI, for example the custom scorer menu in the playground, allow you to define functions. You can also
bundle them in your code and push them to Braintrust with braintrust push
and braintrust eval --push
. Technically speaking,
functions are a generalization of prompts and code functions, so when you define a custom prompt, you are technically defining
a "prompt function".
Every function supports a number of common features:
- Well-defined parameters and return types
- Streaming and non-streaming invocation
- Automatic tracing and logging in Braintrust
- Prompts can be loaded into your code in the OpenAI argument format
- Prompts and code can be easily saved and uploaded from your codebase
See the API docs for more information on how to create and invoke functions.
Sandbox
Functions are executed in a secure sandbox environment. If you are self-hosting Braintrust, then you must:
- Set
EnableQuarantine
totrue
in the Cloudformation stack - Set
ALLOW_CODE_FUNCTION_EXECUTION
to1
in the Docker configuration
If you use our managed AWS stack, custom code runs in a quarantined-VPC in lambda functions which are sandboxed and isolated from your other AWS resources. If you run via Docker, then the code runs in a sandbox but not a virtual machine, so it is your responsibility to ensure that malicious code is not uploaded to Braintrust.
For more information on the security architecture underlying code execution, please reach out to us.