Build evaluation datasets

​

Why use datasets

• Integrated: Use directly in evaluations, explore in playgrounds, and populate from production.
• Versioned: Every change is tracked, so experiments can pin to specific versions.
• Scalable: Stored in a modern data warehouse without storage or performance limits.
• Secure: Self-hosted deployments keep data in your warehouse.

​

Create datasets

• UI
• SDK

​

Upload CSV

The fastest way to create a dataset is uploading a CSV file:

Records with an id key are automatically deduplicated by their id value.

​

Generate with Loop

Ask Loop to create a dataset based on your logs or specific criteria:Example queries:

• “Generate a dataset from the highest-scoring examples in this experiment”
• “Create a dataset with the most common inputs in the logs”

​

Add records manually

Once you’ve created a dataset, add or edit records directly in the UI:

Datasets are created automatically when you initialize them:

TypeScript

Report incorrect code

Copy

Ask AI

import { initDataset } from "braintrust";

async function main() {
  const dataset = initDataset("My App", { dataset: "My New Dataset" });
  console.log("Dataset created:", dataset);
}

main();

​

Insert records

Add records programmatically with insert():

TypeScript

Report incorrect code

Copy

Ask AI

for (let i = 0; i < 10; i++) {
  const id = dataset.insert({
    input: { question: `What is ${i}?` },
    expected: { answer: `${i} is a number` },
    metadata: { category: i % 2 === 0 ? "even" : "odd" },
  });
  console.log("Inserted record with id", id);
}

​

Log from production

Track user feedback from your application:

TypeScript

Report incorrect code

Copy

Ask AI

import { initDataset, Dataset } from "braintrust";

class MyApplication {
  private dataset: Dataset | undefined = undefined;

  async initApp() {
    this.dataset = await initDataset("My App", { dataset: "logs" });
  }

  async logUserExample(
    input: any,
    expected: any,
    userId: string,
    thumbsUp: boolean,
  ) {
    if (this.dataset) {
      this.dataset.insert({
        input,
        expected,
        metadata: { userId, thumbsUp },
      });
    }
  }
}

​

Dataset structure

• input: Data to recreate the example in your application (required).
• expected: Ideal output or ground truth (optional but recommended for evaluation).
• metadata: Key-value pairs for filtering and grouping (optional).

​

View and edit datasets

• Filter and search records
• Create custom columns to extract nested values
• Edit records inline
• Copy records between datasets
• Delete individual records or entire datasets

​

Create custom columns

​

Label datasets

​

Define schemas

• Validation: Ensure records conform to your structure.
• Form-based editing: Edit records with intuitive forms instead of raw JSON.

​

Infer from data

1. Open the schema editor for a field.
2. Click Infer schema.
3. The schema generates from the first 100 records.

​

Enable enforcement

• New records must conform or show validation errors.
• Existing non-conforming records display warnings.
• Form editing validates input automatically.

​

Read and filter datasets

const dataset = initDataset("My App", { dataset: "My Existing Dataset" });

// Loads in batches to avoid memory issues
for await (const row of dataset) {
  console.log(row);
}

const dataset = initDataset("My App", {
  dataset: "My Dataset",
  _internal_btql: {
    filter: { btql: "metadata.category = 'premium'" },
    sort: [{ expr: { btql: "created" }, dir: "desc" }],
    limit: 100,
  },
});

​

Track performance

​

View experiment runs

1. Go to your dataset page.
2. In the right panel, select Runs.
3. Review performance metrics across experiments.

​

Analyze per-row performance

1. Select a row in the dataset table.
2. In the right panel, select Runs.
3. Review the row’s metrics across experiments.

• Consistently low scores suggest ambiguous expectations.
• Failures across experiments indicate edge cases.
• High variance suggests instability.

​

Multimodal datasets

• Image URLs (most performant) - Keep datasets lightweight with external image references.
• Base64 (least performant) - Encode images directly in records.
• Attachments (easiest to manage) - Store files directly in Braintrust.
• External attachments - Reference files in your own object stores.

import { Attachment, initDataset } from "braintrust";
import path from "node:path";

async function createPdfDataset(): Promise<void> {
  const dataset = initDataset({
    project: "Project with PDFs",
    dataset: "My PDF Dataset",
  });
  for (const filename of ["example.pdf"]) {
    dataset.insert({
      input: {
        file: new Attachment({
          filename,
          contentType: "application/pdf",
          data: path.join("files", filename),
        }),
      },
    });
  }
  await dataset.flush();
}

createPdfDataset();

​

Use in evaluations

import { initDataset, Eval } from "braintrust";
import { Levenshtein } from "autoevals";

Eval("Say Hi Bot", {
  data: initDataset("My App", { dataset: "My Dataset" }),
  task: async (input) => {
    return "Hi " + input;
  },
  scores: [Levenshtein],
});

​

Next steps

• Configure human review to label datasets.
• Run evaluations using your datasets.
• Use the Loop to generate and optimize datasets.
• Read the SQL reference for advanced filtering.