Skip to main content
Applies to:
  • Plan -
  • Deployment -

​
Summary

Issue: Running bt eval --dev <file> and connecting to the Playground causes gateway calls to fail with AI_APICallError: Not Found when the AI SDK client reads BRAINTRUST_API_KEY. Cause: bt eval --dev replaces BRAINTRUST_API_KEY with a short-lived Playground session token, which the Braintrust gateway does not accept as a valid credential. Resolution: Pass your real API key to the gateway client via a separate environment variable that bt eval --dev does not overwrite.

​
Environment variables injected by bt eval --dev

When the Playground connects to your local dev server, the following variables are overwritten in your process environment:
VariableValue injected
BRAINTRUST_API_KEYShort-lived Playground session token
BRAINTRUST_API_URLYour org’s API URL
BRAINTRUST_APP_URLDev server app URL
BRAINTRUST_ORG_NAMEOrg you connected from
BRAINTRUST_DEFAULT_PROJECTProject from --project flag or config
The session token injected into BRAINTRUST_API_KEY carries your identity but is not accepted by the Braintrust gateway (gateway.braintrust.dev). Only real API keys (sk-...) or service tokens (bt-st-...) authenticate successfully to the gateway.

​
Resolution steps

​
Step 1: Set a separate env var for your real API key

Before launching the dev server, set an additional variable in the same shell: 
export BRAINTRUST_API_KEY_OVERRIDE=<your-real-api-key>
bt eval --dev <eval-file>

​
Step 2: Update your gateway client to read the override first

import { createOpenAI } from "@ai-sdk/openai";

const btGateway = createOpenAI({
  baseURL: process.env.OPENAI_BASE_URL ?? "https://gateway.braintrust.dev/v1",
  apiKey: process.env.BRAINTRUST_API_KEY_OVERRIDE ?? process.env.BRAINTRUST_API_KEY,
});
The fallback to BRAINTRUST_API_KEY ensures the client still works in non-dev contexts where the override is not set.

​
Notes

  • A 404 Not Found from the AI SDK indicates a bad route or unaccepted credential at the gateway — not a missing org permission. A permission error would surface as 401 or 403.
  • Any env var name not in the injected list above can serve as the override. Common choices: BRAINTRUST_API_KEY_OVERRIDE, BT_API_KEY, BT_GATEWAY_KEY.
  • This workaround is required any time your gateway client reads BRAINTRUST_API_KEY directly in dev mode.