Skip to main content
Applies to:
  • Plan -
  • Deployment -

​
Summary

Issue: Logs stop flowing after a Terraform apply, even when the apply appeared routine (e.g., a provider version upgrade). No error is surfaced to the user. Cause: A Braintrust platform update bundled into the Terraform apply adds BRAINSTORE_ASYNC_SCORING_OBJECTS and BRAINSTORE_LOG_AUTOMATIONS_OBJECTS to the Brainstore launch template user_data; once instances cycle in with these vars set, the index optimization and WAL compaction loops require a valid bt_data_plane_service_token, which was never provisioned. Resolution: Provision the bt_data_plane_service_token in Braintrust settings.

​
Resolution steps

​
Step 1: Confirm the env vars are present

SSH into any running Brainstore EC2 instance and inspect the environment file: 
cat /etc/brainstore.env
If either of the following lines appears, the service token is required: 
BRAINSTORE_ASYNC_SCORING_OBJECTS=all
BRAINSTORE_LOG_AUTOMATIONS_OBJECTS=all
These vars are injected via the launch template user_data, not surfaced as inputs in main.tf. A pending platform update adds them automatically during a Terraform apply. In the Terraform plan output, this appears as user_data = (sensitive value) with a change pending — it may not be obvious that a platform update is included.

​
Step 2: Provision the service token

  1. Go to Settings → Service Tokens
  2. Click Create next to the data plane manager token entry (bt_data_plane_service_token)
The token is stored in RDS. It survives instance cycling, Auto Scaling events, and future deploys. You do not need to reprovision it unless you provision a completely new data plane from scratch.

​
Step 3: Verify logs are flowing

After creating the token, confirm logs resume flowing in your Braintrust data plane. No instance restart or Terraform apply is required.

​
Notes

  • EC2 instance replacement does not require regenerating the service token.
  • Terraform apply runs that include a user_data change on Brainstore launch templates may silently bundle a Braintrust platform update. Review user_data diffs carefully even when the primary change appears minor.
  • To check which env vars are active on running instances at any time, inspect /etc/brainstore.env directly on the EC2 instance.