Export to cloud storage

Documentation Index

Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt

Use this file to discover all available pages before exploring further.

​

How exports work

• Queries recent log data.
• Writes the result to your cloud storage bucket as either JSON Lines or Parquet, organized in a Hive-partitioned layout that warehouses and query engines can read.
• Advances its cursor, so that the same data isn’t re-exported on the next run.
• Schedules the next run: immediately if more data remains, otherwise roughly one interval from when the current run finished.

​

Create an export

# The example below exports trace summaries. To export spans or a custom
# SQL (or BTQL) query, swap export_definition for one of:
#   { "type": "log_spans" }
#   { "type": "btql_query",
#     "btql_query": "select: id, input, output from: project_logs('PROJECT_ID') filter: scores.correctness < 0.5" }
curl --request POST \
  --url https://api.braintrust.dev/v1/project_automation \
  --header "Authorization: Bearer $BRAINTRUST_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "project_id": "PROJECT_ID",
    "name": "Hourly logs export",
    "config": {
      "event_type": "btql_export",
      "export_definition": { "type": "log_traces" },
      "export_path": "s3://my-bucket/braintrust/logs",
      "format": "parquet",
      "interval_seconds": 3600,
      "credentials": {
        "type": "aws_iam",
        "role_arn": "arn:aws:iam::123456789012:role/braintrust-export",
        "external_id": "EXTERNAL_ID"
      }
    }
  }'

curl --request POST \
  --url https://api.braintrust.dev/brainstore/automation/reset-cursors \
  --header "Authorization: Bearer $BRAINTRUST_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "automation_id": "AUTOMATION_ID",
    "object_id": "project_logs:PROJECT_ID",
    "start_xact_id": "0"
  }'

​

Export data types

​

Identity and access management (IAM)

​

Cloud folder structure

{prefix}/
  date=2026-04-19/
    c1f1e5a8-....jsonl.gz
    9de8b4a0-....jsonl.gz
  date=2026-04-20/
    42d0fe3c-....jsonl.gz

• Row order - Rows within a file are sorted ascending by _xact_id, an internal ID assigned at write time. Rows committed together share an _xact_id, and newer writes get higher values. For Logs (spans) exports, this does not imply trace grouping or parent-before-child ordering. To get trace-grouped output, sort in the consuming application.
• File splits - Files split when the date changes or when the file reaches an internal size target. All rows for a given _xact_id always land in the same file, so the exporter writes past the size target when needed to keep them together. File names are random UUIDs to avoid collisions.
• Formats - JSON Lines files are gzipped (.jsonl.gz). Parquet files are ZSTD-compressed (.parquet).

​

Export status and history

1. Go to Settings > Data management.
2. To open the Export status dialog, click the status icon next to your export.
3. Review run history, row counts, bytes written, and duration. These values are cumulative since the export was created or last rewound. Byte counts represent compressed, on-disk sizes in the storage bucket.

​

Rewind to a specific date

1. Open the Export status dialog and click Rewind….
2. In the rewind panel, pick a date and time. Note that date=YYYY-MM-DD/ folders in cloud storage are UTC-based, so they may not match the local time you entered.
3. Click Rewind to selected time to rewind to that moment, or Re-export all data to rewind to the beginning.

• The next run starts immediately, and then the export cadence resumes (with the next run scheduled one interval later).
• Exports are incremental. Runs happen back-to-back while there’s a backlog, with each run continuing from where the previous one stopped. The normal interval cadence resumes once caught up.
• Existing files are kept. Rewind doesn’t delete, modify, or replace files already in your bucket. Subsequent runs write fresh UUID-named files alongside the old ones with overlapping rows, so deduplicate downstream on id or span_id, not on filename.
• Deleted logs can’t be re-exported. Rewinding past your retention cutoff has no effect on logs your retention policy already removed.
• Totals reset. The All runs values in the Export status dialog (rows, data, duration) reset to zero after a rewind and rebuild as new runs land.

​

Troubleshooting

• Ensure you’re on data plane v1.1.27 or later.
• In the Export status dialog, click Rewind… and choose an earlier point in time to re-process from.
• If problems persist, create a new trace export automation.

• Verify the IAM role ARN is correct.
• Check the trust policy includes the correct external ID.
• Ensure the S3 policy grants the required permissions on the target bucket and prefix.
• Confirm the bucket and prefix exist.

• Verify that the service account email pasted into the UI matches the one you created.
• Confirm the service account is listed in brainstore_impersonation_targets in your Braintrust data plane Terraform module, and that Terraform was applied after adding it. See Service account impersonation.
• Confirm the Brainstore service account has the Service Account Token Creator role on your export service account. Braintrust’s Terraform module sets this automatically; if you configured IAM manually, verify it under the export service account’s Principals with access tab.
• Confirm your service account has roles/storage.objectAdmin on the target bucket.
• Allow a few minutes for IAM changes to propagate.

​

Migrate exports to data plane v2.1

• Google Cloud Storage as an export destination (self-hosted GCP only).
• Specify a start-from date when creating an export.
• Rewind lets you re-export historical data. After a rewind, the same row can appear in your bucket more than once; deduplicate downstream on id, keeping the row with the highest _xact_id.
• Dataset and experiment export is no longer supported. To export an experiment or dataset, use SQL or the REST API instead.
• File paths use a Hive-partitioned layout: {prefix}/date=YYYY-MM-DD/ instead of {prefix}/YYYY-MM-DD/. The date is now the row’s transaction date, not the date the export ran. Old-layout files stay in place; scripts matching the old shape will miss new files.
• Files split by size rather than by a 100,000-row cap. Historical catch-up spreads across many date= partitions instead of piling into the current run’s folder, so consumers watching only today’s folder will miss backfill output.
• API change for programmatic creation. The second registration call is now POST /brainstore/automation/reset-cursors instead of POST /automation/cron. Scripts that create exports via the API need to be updated.

​

Limitations

• For self-hosted deployments, Rewind, start-from date, and Google Cloud Storage destinations require data plane v2.1+ (not yet available for self-hosted customers) and must be explicitly enabled through your Terraform module or Helm chart. See Migrate exports to data plane v2.1.
• Only log data can be exported.
• Braintrust-hosted customers can only export their data to AWS S3.
• Self-hosted customers can only export data to the cloud associated with their data plane (AWS S3, and Google Cloud Storage once data plane v2.1 is available for self-hosted customers). Azure Blob Storage is not supported.
• Braintrust’s export writer does not use VPC endpoints or Private Service Connect. Traffic uses the cloud’s public object-store endpoints.

​

Next steps

• Configure data retention to manage storage costs and comply with regulations
• Set up alerts to monitor data quality
• View logs to understand what gets exported
• Export data via the API for one-time exports