Skip to main content
Run the Braintrust Gateway in your data plane to serve model requests and cache completions on your own infrastructure, using provider credentials managed in Braintrust.
Braintrust Gateway self-hosting is in public preview and can change before reaching general availability.

​
Traffic and data handling

Running the Gateway yourself keeps LLM traffic on your own infrastructure: requests egress directly from your VPC to providers, completions are cached (encrypted) in the data plane’s Redis, and internal features like Topics route through it too. When the Gateway is disabled, the data plane does not fall back to the Braintrust-hosted Gateway. The API service calls providers directly. On AWS, quarantine LLM calls route through your deployment’s AI proxy on Terraform module v6.5.0 and later. The Loop runtime does the same on v6.5.2 and later. See quarantine routing for the network paths and PrivateLink setup.
Even with the Gateway enabled, two paths still leave your data plane:
  • Built-in models are served by Braintrust, so requests for them reach the Braintrust-hosted Gateway at gateway.braintrust.dev rather than your own. Self-hosted organizations have built-in models disabled by default and opt in explicitly.
  • Operational telemetry (status, metrics, usage, and optional logs and traces) can still be sent to Braintrust’s control plane, the same as the rest of the data plane. See Telemetry and data retention. This is monitoring traffic, not LLM traffic.

​
Deploy and enable the Gateway

For Terraform module versions before v6.7.2 and Helm chart versions before v6.17.0, before you deploy the Gateway, contact Braintrust to enable it for your organization. Newer module versions do not need this enablement.
Terraform module v7.0.0 and later deploys and enables the Gateway by default. It runs as an ECS Fargate service behind an internal load balancer. To upgrade from v6, follow Upgrade to Terraform module v7.Three variables control the Gateway, so you can provision the infrastructure before routing traffic to it:
  • create_ai_gateway creates the private Gateway infrastructure (an internal ALB and the Gateway ECS service).
  • enable_ai_gateway sets GATEWAY_URL on the API ECS service, and on the APIHandler and AI Proxy Lambda functions when they exist, so internal data plane traffic routes through the Gateway. It requires create_ai_gateway.
  • use_private_ai_gateway_origin points the public CloudFront /v1/proxy endpoint at the private Gateway (a VPC origin on its internal load balancer). It requires create_ai_gateway.
All three variables default to true on v7. To run without the Gateway, set all three to false. Setting only create_ai_gateway = false fails the plan.An enabled Gateway route requires ai_gateway_min_capacity of at least 1. Terraform waits for the Gateway service to reach steady state before routing the API, Loop runtime, CloudFront, or quarantine PrivateLink traffic to it. If the initial deployment fails, those callers keep their previous configuration.
Requires Terraform module v6.5.0 or later, which keeps quarantine LLM calls in your account. All three Gateway variables default to false on v6.Existing deploymentApply each step separately and verify it before continuing:
  1. Provision the infrastructure without changing traffic and apply:
  2. Route internal traffic through the Gateway. Once the Gateway is healthy, set enable_ai_gateway = true and apply:
    Public /v1/proxy traffic still uses its existing origin.
  3. Cut over the public endpoint and apply:
    CloudFront then routes /v1/proxy and /v1/proxy/* to the private Gateway.
New deploymentOn a new v6 deployment, you can set create_ai_gateway and enable_ai_gateway together in the first apply, then cut over the public endpoint with use_private_ai_gateway_origin once the Gateway is healthy.

​
Size and upgrade the Gateway

  • Task size: Set ai_gateway_cpu and ai_gateway_memory.
  • Autoscaling: Set ai_gateway_min_capacity and ai_gateway_max_capacity. An enabled Gateway route requires a minimum capacity of at least 1.
  • Load balancer access: Set ai_gateway_authorized_security_groups to allow additional security groups. The API and Brainstore security groups are authorized automatically.
The Terraform module pins the Gateway image to its data plane release. Upgrade the module to upgrade the Gateway. Leave ai_gateway_version_override unset unless Braintrust instructs you to pin a specific tag.

​
Restrict organization access

Data plane v2.21.0 and later enforces organization restrictions on Gateway requests and AI provider credentials. This version ships in AWS Terraform module v7.1.0 and Helm chart 6.26.0.
Terraform module v7.1.0 and Helm chart 6.26.0 do not copy the API’s organization restrictions to the Gateway. Configure the Gateway separately to restrict its requests.
Set these environment variables on the Gateway:
  • ALLOWED_ORG_IDS: A comma-separated list of organization UUIDs to allow. Defaults to empty. When ORG_NAME names a specific organization, that organization and its credentials are already allowed by name, so list only additional organizations here.
  • ORG_NAME: An organization name to allow in addition to the listed IDs. Defaults to *. When set to * or empty, only ALLOWED_ORG_IDS restricts access. If that list is also empty, all organizations are allowed.
Apply the settings: On AWS, add string entries to the module’s ai_gateway_extra_env_vars map. On GCP or Azure, add name and string value entries to Helm’s aiGateway.extraEnvVars list. Apply the configuration to redeploy the Gateway. The Gateway checks authentication and provider credentials separately:
  • Authentication: Any organization associated with the API key can satisfy the restriction, even if a different organization was selected for the request. Otherwise, the Gateway returns HTTP 403: Organization is not allowed to use this data plane.
  • Provider credentials: Credentials owned by disallowed organizations are filtered out. If all returned credentials are filtered out, the Gateway returns HTTP 403: No provider secrets are allowed for this deployment.

​
Set the Gateway’s API endpoint

Set OVERRIDE_BRAINTRUST_API_URL when the Gateway needs to reach the API at a different URL, such as an internal endpoint. This requires data plane v2.21.0, shipped in AWS Terraform module v7.1.0 and Helm chart 6.26.0.
The override applies to every organization served by the Gateway. Use an endpoint that serves all intended organizations and is reachable from the Gateway. An incorrect endpoint can break project registration, project-level secret fetching, trace logging, and token-budget queries. API-key login and organization-secret fetching use BRAINTRUST_APP_URL.
  1. Set the API base URL on the Gateway:
    • AWS: Add OVERRIDE_BRAINTRUST_API_URL as a string entry in the module’s ai_gateway_extra_env_vars map.
    • GCP or Azure: Add an entry with name: OVERRIDE_BRAINTRUST_API_URL and the URL as its string value in Helm’s aiGateway.extraEnvVars list.
  2. Apply the configuration to redeploy the Gateway.
The override takes precedence over BRAINTRUST_API_URL and organization-specific API URLs, including cached ones. It applies to project registration, project-level secrets, trace logging, and token-budget queries. API-key login and organization-secret fetching use BRAINTRUST_APP_URL. The override is unset by default in Terraform module v7.1.0 and Helm chart 6.26.0. Without it, calls that resolve an organization’s API URL use that URL, falling back to BRAINTRUST_API_URL. Helm’s aiGateway.braintrustApiUrl sets the default URL, not an authoritative override. It defaults to an empty string, which uses the in-cluster API service URL. This setting does not create a private API endpoint or change client connectivity. Configure Gateway organization restrictions separately. This section applies to AWS deployments. By default, LLM calls from user-authored code in the quarantine environment reach the Gateway through the AI proxy Lambda function URL. On Terraform module v7.0.0 and later with enable_ecs_api = true, that function doesn’t exist, so the calls go through the quarantine VPC’s NAT gateway to the deployment’s CloudFront /v1/proxy URL, and CloudFront serves them from whichever origin /v1/proxy uses:
  • If you restrict CloudFront with a WAF or a source IP allowlist, allow the quarantine VPC’s NAT gateway address.
  • With use_global_ai_gateway_origin = true, these calls leave your data plane for the Braintrust-hosted Gateway.
On Terraform module v6.7.0 and later, route these calls over AWS PrivateLink to keep them inside your VPCs:
The module connects quarantine to the Gateway through a network load balancer, VPC endpoint service, and interface endpoint, then configures the API ECS service to use that endpoint. This requires create_ai_gateway and enable_quarantine_vpc. If quarantine is disabled and you haven’t set quarantine_proxy_url, Terraform fails rather than silently falling back.

​
Use existing VPCs

On Terraform module v7.1.0 and later, the module creates the PrivateLink path whether it manages each VPC or you supply it with existing_vpc_id or existing_quarantine_vpc_id. When you supply a VPC:
  • Both VPCs must be in the same AWS account and region as the provider.
  • The three private subnets in each VPC must be distinct and in distinct availability zones, and each quarantine subnet’s availability zone must also have a main VPC private subnet. Terraform validates subnet membership and availability zones.
  • The module doesn’t change your routes, network ACLs, or DNS. DNS in the quarantine VPC must resolve the endpoint hostname, and network ACLs must allow the connection and its return traffic.

​
Use your own endpoint

On earlier module versions, the module creates no PrivateLink resources if you supply either VPC. Create the network load balancer, endpoint service, and quarantine interface endpoint yourself, then point the deployment at your endpoint with quarantine_proxy_url:
quarantine_proxy_url always takes precedence over the automatically selected URL. The quarantine_proxy_url output reports the value the deployment ended up using. When use_global_ai_gateway_origin is true, use_private_gateway_quarantine_proxy has no effect and no PrivateLink resources are created.
The network load balancer and interface endpoint are billed separately, even when quarantine_proxy_url routes traffic elsewhere. To use your own endpoint without these extra resources, set use_private_gateway_quarantine_proxy = false. If you supply either VPC and set both options, make this change before upgrading to v7.1.0 or later.When use_private_gateway_quarantine_proxy is false, quarantine LLM calls reach the Gateway through the AI proxy Lambda function URL, or through the CloudFront /v1/proxy URL on v7.0.0 and later when enable_ecs_api is true. To send them somewhere else, such as the Braintrust-hosted Gateway, set quarantine_proxy_url.
If you use the module-created path with your own VPCs, set use_private_gateway_quarantine_proxy to false before you downgrade to a module version earlier than v7.1.0. Before you turn the option off, make sure quarantine can reach the fallback URL.