Skip to main content
Tool functions in Braintrust allow you to define general-purpose code that can be invoked by LLMs to add complex logic or external operations to your workflows. Tools are reusable and composable, making it easy to iterate on assistant-style agents and more advanced applications. You can create tools in TypeScript or Python and deploy them across the UI and API via prompts.

​
Create a tool

Currently, you must define tools via code and push them to Braintrust with braintrust push. To define a tool, use project.tool.create and pick a name and unique slug. Then push the tool to Braintrust with braintrust push.
  • TypeScript
  • Python
calculator.ts
import * as braintrust from "braintrust";
import { z } from "zod";

const project = braintrust.projects.create({ name: "calculator" });

project.tools.create({
  handler: ({ op, a, b }) => {
    switch (op) {
      case "add":
        return a + b;
      case "subtract":
        return a - b;
      case "multiply":
        return a * b;
      case "divide":
        return a / b;
    }
  },
  name: "Calculator method",
  slug: "calculator",
  description:
    "A simple calculator that can add, subtract, multiply, and divide.",
  parameters: z.object({
    op: z.enum(["add", "subtract", "multiply", "divide"]),
    a: z.number(),
    b: z.number(),
  }),
  returns: z.number(),
  ifExists: "replace",
});

npx braintrust push calculator.ts

​
Dependencies

Braintrust will take care of bundling the dependencies your tool needs. In TypeScript, Braintrust uses esbuild to bundle your code and its dependencies together. This works for most dependencies, but it does not support native (compiled) libraries like SQLite. In Python, Braintrust uses uv to cross-bundle a specified list of dependencies to the target platform (Linux). This works for binary dependencies except for libraries that require on-demand compilation. If you have trouble bundling your dependencies, file an issue in the braintrust-sdk repo.

​
Use tools in the UI

Once you define a tool in Braintrust, you can access it through the UI and API. However, the real advantage lies in calling a tool from an LLM. Most models support tool calling, which allows them to select a tool from a list of available options. Normally, it’s up to you to execute the tool, retrieve its results, and re-run the model with the updated context. Braintrust simplifies this process dramatically by:
  • Automatically passing the tool’s definition to the model
  • Running the tool securely in a sandbox environment when called
  • Re-running the model with the tool’s output
  • Streaming the whole output along with intermediate progress to the client

​
View tools in the UI

Available tools are listed on the Tools page. You can run single datapoints right inside the tool to test its functionality.

​
Add tools to a prompt

To add a tool to a prompt, select it in the Tools dropdown in your Prompt window. Braintrust will automatically:
  • Include it in the list of available tools to the model
  • Invoke the tool if the model calls it, and append the result to the message history
  • Call the model again with the tool’s result as context
  • Continue for up to (default) 5 iterations or until the model produces a non-tool result
This example defines a tool that looks up information about the most recent commit in a GitHub repository and pushes it to Braintrust.
  • TypeScript
  • Python
github.ts
import * as braintrust from "braintrust";
import { z } from "zod";

const project = braintrust.projects.create({ name: "github" });

project.tools.create({
  handler: async ({ org, repo }: { org: string; repo: string }) => {
    const url = `https://api.github.com/repos/${org}/${repo}/commits?per_page=1`;
    const response = await fetch(url);

    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }

    const data = await response.json();

    if (data.length > 0) {
      return data[0];
    } else {
      return null;
    }
  },
  name: "Get latest commit",
  slug: "get-latest-commit",
  description: "Get the latest commit in a repository",
  parameters: z.object({
    org: z.string(),
    repo: z.string(),
  }),
  ifExists: "replace",
});

npx braintrust push github.ts
Once the command completes, the tool is listed in the Library’s Tools tab. Tool code in library Then, you can add the tool to your prompt and run it.

​
Embed tool calls into a prompt

In addition to selecting from the tool menu to add a tool to a prompt, you can also add a tool call directly from the Assistant or Tool messages within a prompt. To add a tool call to an Assistant prompt, select Assistant from the dropdown menu. Then select the Toggle tool calls icon to add the tool code directly into the prompt editor. You can also select Tool from the dropdown menu to enter a tool call ID, such as {{input.3.function_responses.0.id}}.

​
Structured outputs

Another use case for tool calling is to coerce a model into producing structured outputs that match a given JSON schema. You can do this without creating a tool function, and instead use the Raw tab in the Tools dropdown. Enter an array of tool definitions following the OpenAI tool format: Raw tools Braintrust supports two different modes for executing raw tools:
  • auto returns the arguments of the first tool call as a JSON object. This is the default mode.
  • parallel returns an array of all tool calls including both function names and arguments.
Invoke raw tool
response_format: { type: "json_object" } does not get parsed as a JSON object and will be returned as a string.

​
Use tools in code

You can also attach a tool to a prompt defined in code. This example defines a tool and a prompt that uses it and pushes both to Braintrust.
  • TypeScript
  • Python
github.ts
import * as braintrust from "braintrust";
import { z } from "zod";

const project = braintrust.projects.create({ name: "github" });

const latestCommit = project.tools.create({
  handler: async ({ org, repo }: { org: string; repo: string }) => {
    const url = `https://api.github.com/repos/${org}/${repo}/commits?per_page=1`;
    const response = await fetch(url);

    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }

    const data = await response.json();

    if (data.length > 0) {
      return data[0];
    } else {
      return null;
    }
  },
  name: "Get latest commit",
  slug: "get-latest-commit",
  description: "Get the latest commit in a repository",
  parameters: z.object({
    org: z.string(),
    repo: z.string(),
  }),
});

project.prompts.create({
  model: "gpt-4o-mini",
  name: "Commit bot",
  slug: "commit-bot",
  messages: [
    {
      role: "system",
      content: "You are a helpful assistant that can help with GitHub.",
    },
    {
      role: "user",
      content: "{{{question}}}",
    },
  ],
  tools: [latestCommit],
});

npx braintrust push github.ts
You can also define the tool and prompt in separate files and push them together by pushing the prompt file. Note that the Python interpreter only supports relative imports from within a package, so you must either define the tool in the same file as the prompt or use a package structure.