> ## Documentation Index
> Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Harbor

> Sync Harbor agent evaluation results to Braintrust with the native Harbor job plugin, capturing datasets, experiments, scores, and agent traces

[Harbor](https://harborframework.com/) is a Python framework for running AI agent evaluations in isolated Docker containers. Braintrust's native Harbor job plugin syncs evaluation results (datasets, experiments, scores, and agent traces) to Braintrust after each Harbor run.

<View title="Python" icon="https://img.logo.dev/python.org?token=pk_BdcHD9e5SCW3j1rnJkNyMQ">
  <h2 id="setup-python">
    Setup
  </h2>

  Install Braintrust alongside Harbor. Requires `harbor>=0.16.0` and Python 3.12 or later. Docker must be running for Harbor to execute tasks.

  <Steps>
    <Step title="Install packages">
      ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
      pip install braintrust harbor
      ```
    </Step>

    <Step title="Set environment variables">
      ```bash title=".env" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
      BRAINTRUST_API_KEY=<your-braintrust-api-key>
      OPENAI_API_KEY=<your-model-provider-api-key>
      ```
    </Step>
  </Steps>

  <h2 id="run-evaluations-python">
    Run evaluations
  </h2>

  Pass `--plugin braintrust` to `harbor run` to enable the Braintrust plugin. The plugin is discovered automatically through Harbor's `braintrust` entry point after installation.

  <Note>
    The Harbor integration is activated through the `--plugin braintrust` flag in the `harbor run` CLI, not through `auto_instrument()`. No import or patching is required in your task code.
  </Note>

  ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
  harbor run \
    --path task/ \
    --agent terminus-2 \
    --model openai/gpt-5-mini \
    --job-name my-eval \
    --plugin braintrust \
    --plugin-kwarg project_name=my-project \
    --yes
  ```

  After the run completes, the plugin creates a sync manifest at `jobs/my-eval/braintrust-sync.json`.

  <h3 id="configure-env-vars-python">
    Configure with environment variables
  </h3>

  Set `HARBOR_BRAINTRUST_PROJECT` to avoid repeating `--plugin-kwarg project_name=...` on every run:

  ```bash title=".env" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
  HARBOR_BRAINTRUST_PROJECT=my-project
  ```

  Then run without the `project_name` kwarg:

  ```bash theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
  harbor run \
    --path task/ \
    --agent terminus-2 \
    --model openai/gpt-5-mini \
    --job-name my-eval \
    --plugin braintrust \
    --yes
  ```

  <h2 id="backfill-python">
    Backfill a past job
  </h2>

  To re-sync a completed Harbor job without re-running the evaluation, use `backfill_job`:

  ```python title="backfill.py" theme={"theme":{"light":"github-light","dark":"github-dark-dimmed"}}
  import asyncio
  from pathlib import Path
  from braintrust.integrations.harbor import backfill_job

  asyncio.run(backfill_job(Path("jobs/my-eval"), project_name="my-project"))
  ```

  Backfill uses deterministic dataset and experiment identities, so it reconciles with existing Braintrust data instead of creating duplicates.

  <h2 id="what-received-python">
    What Braintrust receives
  </h2>

  After each Harbor evaluation run, Braintrust receives:

  * **A managed dataset** with the task cases used in the evaluation.
  * **An experiment row** for each trial, with the agent's final output as the experiment output and the Harbor task's reference answer as the expected output.
  * **Scores** derived from Harbor verifier rewards, mapped to Braintrust score fields.
  * **Agent traces** (when trajectory tracing is enabled), showing the agent's tool calls and model turns using Harbor's ATIF format.

  <h2 id="resources-python">
    Resources
  </h2>

  * [Harbor documentation](https://harborframework.com/docs)
  * [Harbor plugin example](https://github.com/braintrustdata/braintrust-sdk-python/tree/main/examples/harbor)
  * [Braintrust Python SDK reference](/docs/sdks/python/versions/latest)
</View>