Configure sustained score alerts with Time window alerts

Applies to:

Plan -
Deployment -

Summary

Use a Time window alert to notify when an aggregate score, error rate, or other scalar SQL calculation crosses a threshold over a time window. Time window alerts are the built-in path for sustained score alerts. Use webhook-based external aggregation only when Time window alerts are unavailable in your deployment or when you need custom paging logic that the built-in alert does not express.

What is happening

Log alerts match individual events in evaluation batches. They work well for single-row conditions, but a single outlier can trigger a noisy alert when you care about an aggregate or sustained trend. A Time window alert evaluates one SQL calculation per window and compares the numeric result to a threshold, so it can alert on conditions like average score, p90 score, count, or percentage below a threshold.

Fix or suggestion

Option 1: create a Time window alert

Action: create a Time window alert that computes the aggregate inside Braintrust and notifies you when the result crosses your threshold. For sustained score alerts, configure the calculation to return one numeric value for each evaluated window. Then set Window length (minutes) to the aggregation window, set Alert trigger to your threshold comparison, and use Advanced settings for persistence, recovery, no-data, late-data, schedule, and repeat notification behavior. Example SQL calculation for average factuality:

SELECT AVG(scores.factuality) AS avg_factuality
FROM project_logs('<PROJECT_ID>')
WHERE scores.factuality IS NOT NULL
  AND metadata.environment = 'prod'

For self-hosted deployments, Time window alerts require data plane v2.10.0 or later.

Option 2: use webhook-based external aggregation

Action: use a log alert webhook as a trigger, then query or aggregate data in your own service before paging. Use this fallback when Time window alerts are not available in your deployment, or when your paging policy needs custom state, enrichment, or routing outside the built-in alert configuration.

Create a log alert whose SQL filter captures candidate breaches, for example scores.factuality IS NOT NULL AND scores.factuality < 0.8 AND metadata.environment = 'prod'. Set the action to Webhook and point it at your aggregator endpoint.
Use the webhook payload as a trigger. The alert webhook payload is fixed. It includes the alert metadata, count, time window, and related logs URL, but not every matching log row or score value.
In your aggregator, use the alert time window and the same SQL filter to query Braintrust, or consume the same trace data from your own pipeline.
Compute the windowed condition you care about, such as percentage below threshold, average score, p90, or consecutive count, and only page if that aggregate crosses your threshold.

Minimal Python pseudocode (conceptual):

THRESHOLD = 0.8
PCT_REQUIRED = 0.5  # 50 percent of events

def on_webhook_event(payload):
    time_start = payload["details"]["time_start"]
    time_end = payload["details"]["time_end"]

    rows = query_braintrust_logs(
        start=time_start,
        end=time_end,
        filter="scores.factuality IS NOT NULL AND metadata.environment = 'prod'",
    )

    scores = [
        row["scores"]["factuality"]
        for row in rows
        if row.get("scores", {}).get("factuality") is not None
    ]

    if not scores:
        return

    pct_below_threshold = sum(score < THRESHOLD for score in scores) / len(scores)

    if pct_below_threshold >= PCT_REQUIRED:
        page_ops.send_page({
            "time_start": time_start,
            "time_end": time_end,
            "count": len(scores),
            "pct_below_threshold": pct_below_threshold,
        })

Keep the aggregator simple and observable.

How to confirm it worked

For a Time window alert, use Recent evaluation preview to confirm the calculation returns the expected value, then create test logs that cross and recover from the threshold.
For a Webhook action on a log alert, click Test next to the Webhook URL field. Confirm the webhook receives the expected payload.
For a Slack action, testing from the UI is not available. Log an event that matches the filter, then confirm the channel receives the notification.
For external aggregation, send test events that simulate sustained and transient breaches. Confirm your aggregator only pages for sustained cases and remains silent for single outliers. Check aggregator logs and metrics for window counts and the page history.

Notes

See Alerts for full alert configuration reference.
Time window alerts evaluate scalar SQL calculations. Keep the external aggregation pattern when you need behavior outside that model, such as joining Braintrust data with another incident system before deciding whether to page.

​Summary

​What is happening

​Fix or suggestion

​Option 1: create a Time window alert

​Option 2: use webhook-based external aggregation

​How to confirm it worked

​Notes

Summary

What is happening

Fix or suggestion

Option 1: create a Time window alert

Option 2: use webhook-based external aggregation

How to confirm it worked

Notes