Experiment Fetch Limited to 1000 Events - Export All

Applies to:

Summary

Goal: Export complete experiment data beyond the 1000 event limit using BTQL with cursor pagination. Features: BTQL endpoint, cursor-based pagination via x-bt-cursor header, native query format.

Configuration Steps

Step 1: Understand the limitation

The /v1/experiment/{id}/fetch endpoint has a hard limit of 1000 events per request and does not support cursor pagination.

Step 2: Use BTQL endpoint for full export

The /btql endpoint supports cursor-based pagination to retrieve all experiment events.

import requests
import json

def fetch_all_experiment_events(experiment_id, api_key):
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }
    all_events = []
    cursor = None

    while True:
        query = {
            "query": {
                "from": {
                    "op": "function",
                    "name": {"op": "ident", "name": ["experiment"]},
                    "args": [{"op": "literal", "value": experiment_id}]
                },
                "select": [{"op": "star"}],
                "limit": 1000
            },
            "fmt": "jsonl"
        }

        if cursor:
            query["query"]["cursor"] = cursor

        resp = requests.post(
            "https://api.braintrust.dev/btql",
            headers=headers,
            json=query
        )

        # Extract cursor from response header
        cursor = resp.headers.get("x-bt-cursor")

        # Parse JSONL response
        rows = [
            json.loads(line)
            for line in resp.text.strip().split("\n")
            if line.strip()
        ]

        all_events.extend(rows)

        # Stop when no cursor returned
        if not cursor:
            break

    return all_events

# Usage
events = fetch_all_experiment_events(
    "your-experiment-id",
    "your-api-key"
)
# Print example, handle event data however you need from here.
print(f"Total events: {len(events)}")

Step 3: Extract pagination cursor from headers

The pagination cursor is returned in the x-bt-cursor response header, not in the response body.

Step 4: Pass cursor in subsequent requests

Add the cursor to the query.cursor field in subsequent requests until x-bt-cursor header is null.

Key Differences

Feature	/fetch endpoint	/btql endpoint
Max events per request	1000 (hard limit)	1000 (paginated)
Pagination support	No	Yes (cursor-based)
Cursor location	N/A	x-bt-cursor header
Query format	Simple parameters	Native BTQL format

​Summary

​Configuration Steps

​Step 1: Understand the limitation

​Step 2: Use BTQL endpoint for full export

​Step 3: Extract pagination cursor from headers

​Step 4: Pass cursor in subsequent requests

​Key Differences