Skip to main content
Applies to:


​
Summary

Issue: Downloading JSON exports for large experiments from the Braintrust UI fails with a malloc of size 1073741824 failed error or times out during the download process. Cause: The UI attempts to load the entire dataset into memory at once, causing memory allocation failures for experiments with thousands of records. Resolution: Use the Braintrust API or SDK to paginate and export large experiment data programmatically.

​
Resolution Steps

​
Option 1: Export using TypeScript/JavaScript

​
Step 1: Create export script

Create a script that paginates through experiment data in chunks of 200 records. 
import Braintrust from "braintrust";
import fs from "fs";

const BRAINTRUST_API_KEY = "your-api-key";
const EXPERIMENT_ID = "your-experiment-id";
const CHUNK_SIZE = 200;

const client = new Braintrust.APIClient({ apiKey: BRAINTRUST_API_KEY });

async function exportExperiment() {
  let offset = 0;
  let allData = [];

  while (true) {
    const response = await client.experiment(EXPERIMENT_ID).fetch({
      limit: CHUNK_SIZE,
      offset: offset
    });

    if (response.length === 0) break;
    allData.push(...response);
    offset += CHUNK_SIZE;
  }

  fs.writeFileSync("experiment_data.json", JSON.stringify(allData, null, 2));
}

exportExperiment();

​
Step 2: Run the script

Execute the script to generate experiment_data.json in your current directory.

​
Option 2: Export using Python

​
Step 1: Create export script

Use the Python SDK to fetch and export experiment data with pagination. 
import braintrust
import json

BRAINTRUST_API_KEY = "your-api-key"
EXPERIMENT_ID = "your-experiment-id"
CHUNK_SIZE = 200

client = braintrust.init(api_key=BRAINTRUST_API_KEY)

all_data = []
offset = 0

while True:
    response = client.experiment(EXPERIMENT_ID).fetch(
        limit=CHUNK_SIZE,
        offset=offset
    )

    if not response:
        break

    all_data.extend(response)
    offset += CHUNK_SIZE

with open("experiment_data.json", "w") as f:
    json.dump(all_data, f, indent=2)

​
Step 2: Run the script

Execute the script to generate experiment_data.json in your current directory.

​
Option 3: Export to CSV format

​
Step 1: Modify the script to write CSV

Replace JSON output with CSV writer for spreadsheet compatibility. 
import braintrust
import csv

BRAINTRUST_API_KEY = "your-api-key"
EXPERIMENT_ID = "your-experiment-id"

client = braintrust.init(api_key=BRAINTRUST_API_KEY)

with open("experiment_data.csv", "w", newline="") as f:
    writer = None
    offset = 0

    while True:
        response = client.experiment(EXPERIMENT_ID).fetch(limit=200, offset=offset)
        if not response:
            break

        if writer is None:
            writer = csv.DictWriter(f, fieldnames=response[0].keys())
            writer.writeheader()

        writer.writerows(response)
        offset += 200

​
Additional Notes

A feature request exists to improve UI performance for large experiment downloads, including async export with email notification when ready. Until implemented, API-based exports are the recommended approach for datasets with more than 1000 records.