Data management

Data management automations let you export data to S3 and manage data retention for your Braintrust project.

S3 export

S3 export automations allow you to periodically export your Braintrust data to an AWS S3 bucket. This is ideal for archiving data, running offline analysis, or feeding data into data warehouses like Snowflake or Databricks.

Config S3 export automation

• Automation name: A descriptive name for your export automation
• Description (optional): Additional context about the export automation's purpose
• Type: Select S3 export
• Data to export: Choose what data to export
  • Logs (traces): One row per trace, including scores, token counts, cost, and other metrics
  • Logs (spans): One row per span (lower level)
  • Custom BTQL query: Write your own BTQL query to define the exact data to export
• S3 export path: The S3 path to export the results to (for example, s3://your-bucket-name/path/to/export). Once the automation is created, this path cannot be changed
• Role ARN: The ARN of an IAM role you create in AWS. The UI will help you configure this role
• Format: The file format for the exported data. Choose between JSON Lines and Parquet
• Interval: How frequently the automation should run and export data
• Batch size (advanced): The number of items to export in each batch. Defaults to 1,000

Configure AWS for S3 export

The export configuration relies on you creating an IAM role that Braintrust can assume and use to write to your S3 bucket. This role gets assumed with an external ID that includes your organization ID, project ID, and an automatation-specific ID. If you'd like to reuse this role for multiple export automations across your organization, you can use a wildcard, for example. bt:<your organization ID>:*.

Test and run export automations

Before saving or updating an export automation, you can test it to confirm behavior using the Test automation button. Braintrust will attempt to write (and delete) a small test file to your S3 bucket using the configured IAM role.

View export runs and manual triggers

After creating an export automation, you can view its run history and trigger it manually.

You can also trigger the export automation manually, or reset its state, directly from the status modal.

Data retention

Data retention automations allow you to configure objects in your project to be automatically deleted after a configurable time period. This is helpful for managing storage/cost and complying with data privacy regulations.

Configure data retention

• Automation name: An auto-generated name for your retention automation
• Description (optional): Additional context about the automation's purpose
• Type: Select Data retention
• Object type: Target object type for this retention policy. Currently supports logs
• Retention period: The time period in days to retain matching objects. Once objects are older than this time period they will be automatically marked for deletion and purged from Braintrust