Automations

Automations let you trigger actions based on specific events in Braintrust. This makes it easier for you to execute common actions and integrate Braintrust with your existing tools and workflows.

How automations work

Automations work by monitoring events in your project and executing actions when specified conditions are met. At a high level the automation runtime will:

Limitations

Currently, automations only support log events and webhook actions. More event types and actions coming soon.

Creating automations

To create an automation, select Add automation from the automations tab in your project configuration and input the automation name, a BTQL filter, time interval, and webhook URL.

Automation settings

• Name: A descriptive name for your automation
• Description (optional): Additional context about the automation's purpose
• Event type: Currently supports "Log event"
• BTQL filter: Filter logs using BTQL syntax (if empty, matches all logs)
• Interval: How frequently the automation should check for matching events
• Webhook URL: The endpoint that will receive the automation data

Testing automations

Before saving or updating an automation, you can test it to verify it works as expected. The test will trigger the automation as if the initiating event occurred and then run through the BTQL filter, interval, and execute the action on matching rows. If no matching logs are found, you may need to adjust your BTQL filter or interval. Note that your project must have recent matching logs within the automation interval in order for the test call to succeed.

Webhook payload

When an automation is triggered it sends a JSON payload to your webhook URL with the following structure:

{
  "organization": {
    "id": "org_123",
    "name": "your-organization"
  },
  "project": {
    "id": "proj_456",
    "name": "your-project"
  },
  "automation": {
    "id": "c5b32408-8568-4bff-9299-8cdd56979b67",
    "name": "High-Priority Factuality",
    "description": "Alert on factuality scores for logs with priority 0 in metadata",
    "event_type": "logs",
    "btql_filter": "metadata.priority = 0 AND scores.Factuality < 0.9",
    "interval_seconds": 3600,
    "url": "https://braintrust.dev/app/your-organization/p/your-project/configuration/automations?aid=c5b32408-8568-4bff-9299-8cdd56979b67"
  },
  "details": {
    "is_test": false,
    "message": "High-Priority Factuality: 5 logs triggered automation in the last 1 hour",
    "time_start": "2025-05-12T10:00:00.000Z",
    "time_end": "2025-05-12T11:00:00.000Z",
    "count": 5,
    "related_logs_url": "https://braintrust.dev/app/your-organization/p/your-project/logs?search=..."
  }
}