> ## Documentation Index
> Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure sustained score alerts with external aggregation

> Alert when LLM score degradations persist over a time window using webhook-based external aggregation or filter approximations.

export const plans_0 = "Any"

export const deployments_0 = "Any"

export const data_plane_version_0 = undefined

export const use_case_0 = "Use case - Alert when LLM judge score breaches persist over a time window (for example, sustained low factuality for 5 minutes) by using a webhook-based aggregator or configuration approximations"

<Note>
  **Applies to:**

  * Plan - {plans_0}
  * Deployment - {deployments_0}
  * {data_plane_version_0}
  * {use_case_0}
</Note>

## Summary

Chart visuals cannot be used directly to create time-windowed alerts. Braintrust log alerts match events in evaluation batches and cannot require a condition to persist for N minutes. Use a log alert that mirrors your chart and either tune filters/notify-intervals or send alert notifications to a webhook and have an external service query or aggregate the underlying logs before paging.

## What is happening

There is no built-in "sustained for 5 minutes" or sliding-window aggregate (avg/p90 over X minutes) condition. That causes single outliers to generate alerts unless you approximate persistence with stricter filters or external aggregation.

## Fix or suggestion

### Option 1: mirror chart with a log alert (fast, limited)

1. Go to **<Icon icon="settings-2" /> Settings** > [**<Icon icon="bell" /> Alerts**](https://www.braintrust.dev/app/~/configuration/alerts) and create a new Log alert.
   ```sql theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
   scores.factuality IS NOT NULL
   AND scores.factuality < 0.8
   AND environment = 'prod'
   ```
2. Choose an action (Slack or Webhook).
3. Set a notify interval long enough to reduce noise (e.g., 30m).
4. Test the alert from the UI.

<Note>
  This will still fire on a single matching event in an evaluation batch. The notify interval suppresses repeat notifications but does not require persistence.
</Note>

### Option 2: webhook + external 5-minute aggregator (recommended for “sustained”)

1. Create a Log alert whose SQL filter captures candidate breaches, for example `scores.factuality IS NOT NULL AND scores.factuality < 0.8 AND metadata.environment = 'prod'`. Set the action to Webhook and point it at your aggregator endpoint.
2. Use the webhook payload as a trigger. The alert webhook payload is fixed; it includes the alert metadata, count, time window, and related logs URL, but not every matching log row or score value.
3. In your aggregator, use the alert time window and the same SQL filter to query Braintrust, or consume the same trace data from your own pipeline.
4. Compute the windowed condition you care about, such as percentage below threshold, average score, p90, or consecutive-count, and only page if that aggregate crosses your threshold.

Minimal Python pseudocode (conceptual):

```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
THRESHOLD = 0.8
PCT_REQUIRED = 0.5  # e.g. 50% of events

def on_webhook_event(payload):
    time_start = payload["details"]["time_start"]
    time_end = payload["details"]["time_end"]

    rows = query_braintrust_logs(
        start=time_start,
        end=time_end,
        filter="scores.factuality IS NOT NULL AND metadata.environment = 'prod'",
    )

    scores = [
        row["scores"]["factuality"]
        for row in rows
        if row.get("scores", {}).get("factuality") is not None
    ]

    if not scores:
        return

    pct_below_threshold = sum(score < THRESHOLD for score in scores) / len(scores)

    if pct_below_threshold >= PCT_REQUIRED:
        page_ops.send_page({
            "time_start": time_start,
            "time_end": time_end,
            "count": len(scores),
            "pct_below_threshold": pct_below_threshold,
        })
```

Keep the aggregator simple and observable.

## How to confirm it worked

* For Option 1: Use UI Test alert. Confirm the webhook or Slack receives the expected single-match payload.
* For Option 2: Send test events that simulate sustained and transient breaches. Confirm your aggregator only pages for sustained cases and remains silent for single outliers. Check aggregator logs/metrics for window counts and the page history.

## Notes

* See [Alerts](/docs/observe/alerts) for full alert configuration reference.
* Native windowed or aggregate alert conditions are a tracked feature request.