Automations

Automations let you trigger actions based on specific events in Braintrust. This makes it easier for you to execute common actions and integrate Braintrust with your existing tools and workflows.

How automations work

Automations work by monitoring events in your project and executing actions when specified conditions are met. At a high level the automation runtime will:

Automation types

Braintrust currently supports two types of automations. To create one, navigate to the Configuration > Automations tab in your project and select Add automation.

• Webhook: Send a JSON payload to a specified URL.
• S3 export: Export data to an AWS S3 bucket in JSONL or Parquet format.

Webhook automation

Webhook automations send a JSON payload to an endpoint whenever the automation's conditions are met. This is useful for triggering alerts in tools like Slack or PagerDuty, or for kicking off custom workflows.

Settings

• Name: A descriptive name for your automation.
• Description (optional): Additional context about the automation's purpose.
• Event type: Select Log event.
• BTQL filter: Filter logs using BTQL syntax (if empty, matches all logs).
• Interval: How frequently the automation should check for matching events.
• Webhook URL: The endpoint that will receive the automation data.

Webhook payload

When a webhook automation is triggered, it sends a JSON payload to your webhook URL with the following structure:

{
  "organization": {
    "id": "org_123",
    "name": "your-organization"
  },
  "project": {
    "id": "proj_456",
    "name": "your-project"
  },
  "automation": {
    "id": "c5b32408-8568-4bff-9299-8cdd56979b67",
    "name": "High-Priority Factuality",
    "description": "Alert on factuality scores for logs with priority 0 in metadata",
    "event_type": "logs",
    "btql_filter": "metadata.priority = 0 AND scores.Factuality < 0.9",
    "interval_seconds": 3600,
    "url": "https://braintrust.dev/app/your-organization/p/your-project/configuration/automations?aid=c5b32408-8568-4bff-9299-8cdd56979b67"
  },
  "details": {
    "is_test": false,
    "message": "High-Priority Factuality: 5 logs triggered automation in the last 1 hour",
    "time_start": "2025-05-12T10:00:00.000Z",
    "time_end": "2025-05-12T11:00:00.000Z",
    "count": 5,
    "related_logs_url": "https://braintrust.dev/app/your-organization/p/your-project/logs?search=..."
  }
}

S3 export automation

S3 export automations allow you to periodically export your Braintrust data to an AWS S3 bucket. This is ideal for archiving data, running offline analysis, or feeding data into data warehouses like Snowflake or Databricks.

Settings

• Name: A descriptive name for your automation.
• Description (optional): Additional context about the automation's purpose.
• Event type: Select BTQL export.
• Data to export: Choose what data to export.
  • Logs (traces): One row per trace, including scores, token counts, cost, and other metrics.
  • Logs (spans): One row per span (lower level).
  • Custom BTQL query: Write your own BTQL query to define the exact data to export.
• S3 export path: The S3 path to export the results to (for example, s3://your-bucket-name/path/to/export). Once the automation is created, this path cannot be changed.
• Role ARN: The ARN of an IAM role you create in AWS. The UI will help you configure this role.
• Format: The file format for the exported data. Choose between JSON Lines and Parquet.
• Interval: How frequently the automation should run and export data.
• Batch size (advanced): The number of items to export in each batch. Defaults to 1,000.

Configuring AWS for S3 Export

The export configuration relies on you creating an IAM role that Braintrust can assume and use to write to your S3 bucket. This role gets assumed with an external ID that includes your organization ID, project ID, and an automatation-specific ID. You can use a wildcard, for example, bt:<your organization ID>:* if you'd like to reuse this role for multiple automations across your organization.

Testing and running automations

Before saving or updating an automation, you can test it by selecting Test automation.

For webhooks, Braintrust sends a test payload to your URL. For S3 exports, Braintrust attempts to write (and delete) a small test file to your S3 bucket.

The test will trigger the automation as if the initiating event occurred, running it through the BTQL filter on recent data. If no matching logs are found, you may need to adjust your BTQL filter or the automation interval.

Viewing runs and manual triggers

After creating an automation, you can view its run history and trigger it manually.

You can also trigger the automation manually, or reset its state, directly from the status modal.