> ## Documentation Index
> Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Run experiments in CI/CD

> Automate evaluation runs in CI with GitHub Actions or the bt eval CLI to keep regressions off your main branch.

Integrate evaluations into your CI/CD pipeline to catch regressions before they reach production. Run evals on every pull request, gate merges on the results, and report scores back to your team automatically.

## GitHub Actions

Use the [`braintrustdata/eval-action`](https://github.com/braintrustdata/eval-action) to run evaluations on every pull request:

```yaml theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
name: Run evaluations

on:
  pull_request:
    branches: [main]

permissions:
  pull-requests: write
  contents: read

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 24

      - name: Install dependencies
        run: npm install

      - name: Run evals
        uses: braintrustdata/eval-action@v2
        with:
          api_key: ${{ secrets.BRAINTRUST_API_KEY }}
          runtime: node
```

Set `runtime` to `node`, `python`, or `go` to match your project. The action automatically posts a comment with results to the pull request, which requires the `pull-requests: write` permission shown above.

## Other CI systems

For other CI systems, use [`bt eval`](/docs/reference/cli/eval) directly:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
BRAINTRUST_API_KEY=$BRAINTRUST_API_KEY bt eval evals/ --no-input --json
```

Create an API key under **<Icon icon="settings-2" /> Settings** > [**<Icon icon="key-square" /> API keys**](https://www.braintrust.dev/app/~/configuration/org/api-keys) and set it as `BRAINTRUST_API_KEY` in your CI environment. Use `--no-input` to suppress prompts and `--json` for machine-readable output.

Use `--first N` or `--sample N` to run a non-final smoke test on pull requests and reserve the full run for merges:

```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
bt eval tests/ --first 20 --no-input --json    # smoke run on PR, non-final
bt eval tests/ --no-input --json               # full run on merge, final
```

## Customize pass/fail with reporters

When you run an experiment, Braintrust logs results to your terminal, and `bt eval` returns a non-zero exit code if any eval throws an exception. Customize this behavior for CI/CD pipelines to precisely define what constitutes a failure or to report results to different systems.

Define custom reporters using `Reporter()`. A reporter has two functions:

<CodeGroup dropdown>
  ```typescript theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
  import { Reporter } from "braintrust";

  Reporter(
    "My reporter", // Replace with your reporter name
    {
      reportEval(evaluator, result, opts) {
        // Summarizes the results of a single evaluator and returns whatever you
        // want (the full results, a piece of text, or both)
      },

      reportRun(results) {
        // Takes all the results and summarizes them. Return a true or false
        // which tells the process to exit.
        return true;
      },
    },
  );
  ```

  ```python theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
  from braintrust import Reporter

  def report_eval(evaluator, result, opts):
      # Summarizes the results of a single evaluator and returns whatever you
      # want (the full results, a piece of text, or both)
      pass

  def report_run(results):
      # Takes all the results and summarizes them. Return a true or false
      # which tells the process to exit.
      return True

  Reporter(
      "My reporter",  # Replace with your reporter name
      report_eval=report_eval,
      report_run=report_run,
  )
  ```
</CodeGroup>

Any `Reporter` included among your evaluated files will be automatically picked up by the `bt eval` CLI command.

* If no reporters are defined, the default reporter logs results to the console.
* If you define one reporter, it's used for all `Eval` blocks.
* If you define multiple `Reporter`s, specify the reporter name as an optional third argument to the eval function.

## Next steps

* [Run experiments in code](/docs/evaluate/run-in-code) with the SDK
* [Interpret results](/docs/evaluate/interpret-results) from your experiments
* [Compare experiments](/docs/evaluate/compare-experiments) to measure improvements
* [Write scorers](/docs/evaluate/write-scorers) to measure quality