> ## Documentation Index
> Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Alerting on scorer errors and aggregated scores

export const plans_0 = "Any"

export const deployments_0 = "Any"

export const data_plane_version_0 = undefined

export const use_case_0 = "Use case - Detect when a scorer is broken or when the recent average score drops below a threshold by using UI filters, Time window alerts, or an external scheduled SQL check as a fallback"

<Note>
  **Applies to:**

  * Plan - {plans_0}
  * Deployment - {deployments_0}
  * {data_plane_version_0}
  * {use_case_0}
</Note>

## Summary

Use a log alert with a span-level error filter to catch scorer failures: `span_attributes.name = '<Scorer name>' AND error IS NOT NULL`. For aggregated checks, use a **Time window** alert when that alert type is available. For self-hosted deployments, **Time window** alerts require [data plane v2.10.0 or later](/docs/data-plane-changelog). If your deployment does not show **Time window** as an alert type, run a scheduled SQL check externally and notify from there.

## What is happening

Log alerts evaluate individual rows. Any row that matches your filter can trigger an alert. That works well for scorer failures, but per-row score thresholds are noisy when you want an hourly average or another aggregate. **Time window** alerts evaluate a scalar SQL calculation over a window and compare the numeric result to a threshold.

The scorer `error` field is populated when the scorer throws or encounters an error while running. Filtering on that field targets actual scorer failures instead of yet-to-be-processed or empty logs.

## Fix or suggestion

### Option 1: alert on scorer errors (recommended)

Action: create a UI alert that only fires when the scorer produced an error.

* In the alert filter use:

  ```sql theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
  span_attributes.name = 'Feed Density' AND error IS NOT NULL
  ```

* Save the alert and set an appropriate notification interval to avoid repeated notifications.

Why: This targets only spans where the scorer explicitly failed.

### Option 2: aggregated or rolling-window check (recommended when available)

Action: create a **Time window** alert that computes an aggregate over a time window and notifies you if the value crosses your threshold.

* Go to **<Icon icon="settings-2" /> Settings** > [**<Icon icon="bell" /> Alerts**](https://www.braintrust.dev/app/~/configuration/alerts).
* Click <Icon icon="plus" /> **Alert**.
* Select **Time window** as the alert type.
* Configure the calculation with the builder or a SQL query.
* Set **Window length (minutes)** to the aggregation window.
* Set **Alert trigger** to the comparison you want. For example, set value is `<` `0.8`.
* Use **Advanced settings** to configure trigger delay, late-data handling, recovery notifications, evaluation schedule, and renotification.

Example raw SQL calculation:

```sql theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
SELECT AVG(scores."Feed Density") AS avg_score
FROM project_logs('<PROJECT_ID>')
WHERE scores."Feed Density" IS NOT NULL
```

Why: A **Time window** alert computes the aggregate inside Braintrust, compares the scalar result to your threshold, and handles recovery and no-data behavior.

### Option 3: external scheduled aggregate check (fallback)

Action: run a scheduled SQL job that computes an aggregate over a time window and notify externally if it drops below your threshold.

* Example query:

  ```sql theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
  SELECT AVG(scores."Feed Density") AS avg_score, COUNT(*) AS scored_count
  FROM project_logs('<PROJECT_ID>')
  WHERE created >= NOW() - INTERVAL 1 HOUR
    AND scores."Feed Density" IS NOT NULL
  HAVING AVG(scores."Feed Density") < 0.8
  ```

* Schedule this query to run at your desired cadence, such as every 5 to 15 minutes.

* Use your external notifier, such as Slack, PagerDuty, email, or another destination, to send alerts when the query returns a row.

Why: Use this fallback only when **Time window** alerts are not available in your deployment.

### Additional guidance to reduce noise

* Exclude yet-to-be-processed logs by requiring `scores IS NOT NULL` in filters.
* Add a short buffer to the time window if ingestion latency causes transient nulls. For example, ignore the most recent 5 to 10 minutes.
* Tune notification frequency or suppression to avoid alert storms.

## How to confirm it worked

* Error alert check: run the following query to find recent scorer errors, then confirm the alert fires for matching rows.

  ```sql theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
  SELECT id, created, span_attributes, error
  FROM project_logs('<PROJECT_ID>')
  WHERE span_attributes.name = 'Feed Density'
    AND error IS NOT NULL
  LIMIT 5
  ```

* Aggregation check:
  Use the **Recent evaluation preview** in the **Time window** alert to confirm the calculation returns the expected value. If you use the external fallback, run the scheduled aggregate query manually. Confirm `avg_score` is below your threshold and that your external notifier receives the alert.

## Notes

* For self-hosted deployments, **Time window** alerts require [data plane v2.10.0 or later](/docs/data-plane-changelog).