Tools

Tool functions in Braintrust allow you to define general-purpose code that can be invoked by LLMs to add complex logic or external operations to your workflows. Tools are reusable and composable, making it easy to iterate on assistant-style agents and more advanced applications. You can create tools in TypeScript or Python and deploy them across the UI and API via prompts.

Create a tool

Currently, you must define tools via code and push them to Braintrust with braintrust push. To define a tool, use project.tool.create and pick a name and unique slug. Then push the tool to Braintrust with braintrust push.

TypeScript
Python

calculator.ts

import * as braintrust from "braintrust";
import { z } from "zod";

// Get a handle to the project (creates if it doesn't exist)
const project = braintrust.projects.create({ name: "calculator" });

project.tools.create({
  handler: ({ op, a, b }) => {
    switch (op) {
      case "add":
        return a + b;
      case "subtract":
        return a - b;
      case "multiply":
        return a * b;
      case "divide":
        return a / b;
    }
  },
  name: "Calculator method",
  slug: "calculator",
  description:
    "A simple calculator that can add, subtract, multiply, and divide.",
  parameters: z.object({
    op: z.enum(["add", "subtract", "multiply", "divide"]),
    a: z.number(),
    b: z.number(),
  }),
  returns: z.number(),
  ifExists: "replace",
});

npx braintrust push calculator.ts

calculator.py

from typing import Literal

import braintrust
import requests
from pydantic import BaseModel, RootModel

# Get a handle to the project (creates if it doesn't exist)
project = braintrust.projects.create(name="calculator")

class CalculatorInput(BaseModel):
    op: Literal["add", "subtract", "multiply", "divide"]
    a: float
    b: float

class CalculatorOutput(RootModel[float]):
    pass

def calculator(op, a, b):
    match op:
        case "add":
            return a + b
        case "subtract":
            return a - b
        case "multiply":
            return a * b
        case "divide":
            return a / b

project.tools.create(
    handler=calculator,
    name="Calculator method",
    slug="calculator-2",
    description="A simple calculator that can add, subtract, multiply, and divide.",
    parameters=CalculatorInput,  # You can also provide raw JSON schema here if you prefer
    returns=CalculatorOutput,
)

braintrust push calculator.py

Dependencies

Braintrust will take care of bundling the dependencies your tool needs. In TypeScript, Braintrust uses esbuild to bundle your code and its dependencies together. This works for most dependencies, but it does not support native (compiled) libraries like SQLite. In Python, Braintrust uses uv to cross-bundle a specified list of dependencies to the target platform (Linux). This works for binary dependencies except for libraries that require on-demand compilation. If you have trouble bundling your dependencies, file an issue in the braintrust-sdk repo.

Use tools in the UI

Once you define a tool in Braintrust, you can access it through the UI and API. However, the real advantage lies in calling a tool from an LLM. Most models support tool calling, which allows them to select a tool from a list of available options. Normally, it’s up to you to execute the tool, retrieve its results, and re-run the model with the updated context. Braintrust simplifies this process dramatically by:

Automatically passing the tool’s definition to the model
Running the tool securely in a sandbox environment when called
Re-running the model with the tool’s output
Streaming the whole output along with intermediate progress to the client

View tools in the UI

Available tools are listed on the Tools page.

You can run single datapoints right inside the tool to test its functionality.

Add tools to a prompt

To add a tool to a prompt, select it in the Tools dropdown in your Prompt window. Braintrust will automatically:

Include it in the list of available tools to the model
Invoke the tool if the model calls it, and append the result to the message history
Call the model again with the tool’s result as context
Continue for up to (default) 5 iterations or until the model produces a non-tool result

This example defines a tool that looks up information about the most recent commit in a GitHub repository and pushes it to Braintrust.

TypeScript
Python

github.ts

import * as braintrust from "braintrust";
import { z } from "zod";

// Get a handle to the project (creates if it doesn't exist)
const project = braintrust.projects.create({ name: "github" });

project.tools.create({
  handler: async ({ org, repo }: { org: string; repo: string }) => {
    const url = `https://api.github.com/repos/${org}/${repo}/commits?per_page=1`;
    const response = await fetch(url);

    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }

    const data = await response.json();

    if (data.length > 0) {
      return data[0];
    } else {
      return null;
    }
  },
  name: "Get latest commit",
  slug: "get-latest-commit",
  description: "Get the latest commit in a repository",
  parameters: z.object({
    org: z.string(),
    repo: z.string(),
  }),
  ifExists: "replace",
});

npx braintrust push github.ts

github.py

import braintrust
import requests
from pydantic import BaseModel

# Get a handle to the project (creates if it doesn't exist)
project = braintrust.projects.create(name="github")

class Args(BaseModel):
    org: str
    repo: str

def handler(org, repo):
    url = f"https://api.github.com/repos/{org}/{repo}/commits?per_page=1"
    resp = requests.get(url)
    resp.raise_for_status()
    data = resp.json()
    if len(data) > 0:
        return data[0]
    return None

project.tools.create(
    handler=handler,
    name="Get latest commit",
    slug="get-latest-commit",
    description="Get the latest commit in a repository",
    parameters=Args,
)

braintrust push github.py

Once the command completes, the tool is listed in the Library’s Tools tab.

Then, you can add the tool to your prompt and run it.

Embed tool calls into a prompt

In addition to selecting from the tool menu to add a tool to a prompt, you can also add a tool call directly from the Assistant or Tool messages within a prompt.

To add a tool call to an Assistant prompt, select Assistant from the dropdown menu. Then select the Toggle tool calls icon to add the tool code directly into the prompt editor. You can also select Tool from the dropdown menu to enter a tool call ID, such as {{input.3.function_responses.0.id}}.

Structured outputs

Another use case for tool calling is to coerce a model into producing structured outputs that match a given JSON schema. You can do this without creating a tool function, and instead use the Raw tab in the Tools dropdown. Enter an array of tool definitions following the OpenAI tool format:

Braintrust supports two different modes for executing raw tools:

auto returns the arguments of the first tool call as a JSON object. This is the default mode.
parallel returns an array of all tool calls including both function names and arguments.

response_format: { type: "json_object" } does not get parsed as a JSON object and will be returned as a string.

Use tools in code

You can also attach a tool to a prompt defined in code. This example defines a tool and a prompt that uses it and pushes both to Braintrust.

TypeScript
Python

github.ts

import * as braintrust from "braintrust";
import { z } from "zod";

// Get a handle to the project (creates if it doesn't exist)
const project = braintrust.projects.create({ name: "github" });

const latestCommit = project.tools.create({
  handler: async ({ org, repo }: { org: string; repo: string }) => {
    const url = `https://api.github.com/repos/${org}/${repo}/commits?per_page=1`;
    const response = await fetch(url);

    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }

    const data = await response.json();

    if (data.length > 0) {
      return data[0];
    } else {
      return null;
    }
  },
  name: "Get latest commit",
  slug: "get-latest-commit",
  description: "Get the latest commit in a repository",
  parameters: z.object({
    org: z.string(),
    repo: z.string(),
  }),
});

project.prompts.create({
  model: "gpt-4o-mini",
  name: "Commit bot",
  slug: "commit-bot",
  messages: [
    {
      role: "system",
      content: "You are a helpful assistant that can help with GitHub.",
    },
    {
      role: "user",
      content: "{{{question}}}",
    },
  ],
  tools: [latestCommit],
});

npx braintrust push github.ts

commit-bot.py

import braintrust
import requests
from pydantic import BaseModel

# Get a handle to the project (creates if it doesn't exist)
project = braintrust.projects.create(name="github")

class Args(BaseModel):
    org: str
    repo: str

def handler(org, repo):
    url = f"https://api.github.com/repos/{org}/{repo}/commits?per_page=1"
    resp = requests.get(url)
    resp.raise_for_status()
    data = resp.json()
    if len(data) > 0:
        return data[0]
    return None

latest_commit = project.tools.create(
    handler=handler,
    name="Get latest commit",
    slug="get-latest-commit",
    description="Get the latest commit in a repository",
    parameters=Args,
)

project.prompts.create(
    model="gpt-4o-mini",
    name="Commit bot",
    slug="commit-bot",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that can help with GitHub.",
        },
        {
            "role": "user",
            "content": "{{{question}}}",
        },
    ],
    tools=[latest_commit],
)

braintrust push commit-bot.py

You can also define the tool and prompt in separate files and push them together by pushing the prompt file. Note that the Python interpreter only supports relative imports from within a package, so you must either define the tool in the same file as the prompt or use a package structure.

Start

Integrations

Core

Context

Best practices

Reference

Create a tool

Dependencies

Use tools in the UI

View tools in the UI

Add tools to a prompt

Embed tool calls into a prompt

Structured outputs

Use tools in code

Start

Integrations

Core

Context

Best practices

Reference

​Create a tool

​Dependencies

​Use tools in the UI

​View tools in the UI

​Add tools to a prompt

​Embed tool calls into a prompt

​Structured outputs

​Use tools in code

Create a tool

Dependencies

Use tools in the UI

View tools in the UI

Add tools to a prompt

Embed tool calls into a prompt

Structured outputs

Use tools in code