The Braintrust CLI, bt, gives a coding agent with an authenticated shell direct access to your production logs and your evals. That means it can read the trace behind a reported failure, turn it into a test case, and check whether its own fix improves the result.
These five tips cover the commands that make that loop work: bt eval --watch while you edit a prompt, seeded smoke runs for faster CI feedback, bt view and bt sql for latency triage, bt sync plus bt datasets to capture a production failure as an eval case, and bt functions push to keep scorers in your repo.
Start free with Braintrust to run evals and query production logs from your terminal →
If like every other developer, you're spending most of your day talking to a coding agent, you probably know the value of getting context directly to the agent. The best way to get the right context on how your agents are performing on your production traces and local development is to give your agent the context you already have in Braintrust.
The Braintrust CLI, called bt, gives your coding agent access to Braintrust through the terminal. It's a command-line interface that lets an agent with an authenticated shell query your production logs directly. The agent can also run your evals, which means it has a way to check whether a proposed fix improves the answers your application produces.
That changes how you can work through a problem together. If a support assistant gets your return policy wrong, your coding agent can inspect the original conversation as it investigates the cause. You can then work with it on an eval case that captures the correct behavior, giving you a way to test the fix before it reaches another customer.
The same commands work when you run them yourself or put them in CI, so the workflow can carry over from a conversation with your agent into your regular development process. These five tips show how to make that useful in practice.
Before you start
The examples assume you have bt installed and authenticated, with eval files and their dependencies in your project. Adapt the example names and paths to your project before running the commands. bt eval currently supports macOS and Linux.
1. Keep an eval running while you edit
When you're adjusting a prompt, watch mode saves you from rerunning the same command after every edit, so you can keep your attention on how the answers change:
bt eval tests/support.eval.ts --watch
Braintrust reruns the eval when input files change, and you can apply the same behavior to evals across a directory:
bt eval tests/ --watch
If your support assistant misses the sale-item exception in a return policy, start with the eval that captures that case. Keeping it in view gives you feedback on the behavior you want to fix without as much unrelated output to sift through.
While you work on the prompt, keep the dataset and scorer steady so you can tell which changes actually help. If all three change together, a higher score becomes harder to interpret, and you may lose track of what improved the answer. Once you're happy with the result, run the broader suite to check how the change affects your other cases.
2. Use repeatable smoke checks for faster CI feedback
A smoke check helps you get feedback on a pull request while the change is still fresh in your head, without waiting for the whole dataset to finish. You can start with the first 20 examples:
bt eval tests/ --first 20 --no-input --jsonl
The examples you choose matter, though. If your first 20 rows all concern refunds, the check tells you little about shipping questions, so a seeded random sample can give you a selection from across the dataset:
bt eval tests/ --sample 20 --sample-seed 7 --no-input --jsonl
Keeping the dataset and seed the same across revisions lets you compare the same selection of examples, even though the model's answers can still vary.
Because a subset only gives you a partial view, Braintrust labels these runs as non-final. You can use them for early feedback while keeping a full-suite check before release, with scheduled runs to catch issues between releases:
bt eval tests/ --no-input --jsonl
For evals in CI/CD, inject BRAINTRUST_API_KEY through your secret store and install the CLI and eval dependencies. The output flags make this scriptable: --no-input disables prompts, while --jsonl emits a JSON summary per evaluator, including a runMode and isFinal field you can read in a script. You'll also need to decide which results should block a change. By default bt eval exits non-zero only when an eval throws an exception, so a low score does not fail the run on its own. To turn scores into a pass/fail signal, define a custom reporter: Reporter() takes a reportRun function whose boolean return value sets the process exit status.
3. Find the slow requests before guessing at a fix
When a user reports that your agent is slow, the logs can help you narrow down where the time goes. With bt view, you can inspect requests from the last 24 hours that took more than five seconds:
bt view logs --project my-project --window 24h \
--filter "metrics.duration > 5.0"
From there, you can open a trace and inspect its spans, which represent the individual steps within the request. You might find that the model call finishes quickly but the agent calls the same tool four times, giving you a specific behavior to investigate as you work on latency.
Your coding agent can investigate that behavior too. The JSON output from bt sql gives it results it can read directly from the shell:
bt sql --project my-project --non-interactive \
"SELECT * FROM logs WHERE created > now() - interval 1 day AND metrics.duration > 5.0 LIMIT 20" \
--json
With those results available, you can ask your coding agent to investigate the repeated tool calls in your repository. It has a production example to work from as it follows the code responsible for that behavior, which gives its proposed fix a clearer basis.
Including the project and time window also makes the query easy to reuse. A teammate or coding agent can run it without depending on the project you last selected in your terminal.
4. Turn a production failure into a reusable test case
When a user encounters a failure your evals missed, that interaction gives you a concrete example to add to your tests. Use bt sync to pull recent production logs into local NDJSON files for review:
bt sync pull project_logs:my-project --window 24h
Suppose your assistant told a customer they could return a sale item, even though the policy says all sale items are final sale. Your coding agent can help turn that conversation into a dataset row using the input shape your eval already expects. You still need to review the expected answer against the actual policy so the test rewards the correct behavior.
For this example, save the following as regression-cases.json. The input preserves the question and policy context, while expected records the answer the assistant should have given:
[
{
"id": "sale-return-001",
"input": {
"question": "Can I return an item bought on sale?",
"policy": "Full-price items may be returned within 30 days. Sale items are final sale."
},
"expected": "Sale items are final sale and cannot be returned.",
"metadata": {
"failure_type": "policy_exception"
}
}
]
Once you review the row, create a dataset from the file:
bt datasets create production-regressions --project my-project \
--file regression-cases.json
Configure your eval to read this dataset so your coding agent can run the case as it works on a fix. For later additions to the same dataset, use bt datasets update with stable record IDs, supplied through an id field or --id-field, since rows without an ID are rejected on update.
The export supplies the raw material, but selecting a useful failure and establishing the correct answer is what makes it an eval case. Our guide to turning LLM production failures into regression tests goes deeper on choosing examples worth testing.
5. Keep scorers and tools alongside your application code
Keeping a scorer in your repo gives your coding agent access to the rule that judges its changes. If you're improving how your support assistant handles policy exceptions, the agent can read that scorer as it works on the prompt, and any proposed scoring change appears in the same diff for you to review.
Once you're happy with the definition, bt functions push deploys it to Braintrust:
bt functions push src/scorers.py
The CLI discovers and bundles the functions your file registers through the Braintrust SDK. This also works for tools and LLM functions you want to maintain in your repo, with support for Python and TypeScript definitions.
If a function with that slug already exists, the CLI returns an error by default, so you'll need to request a replacement when you're ready to update it:
bt functions push --if-exists replace src/scorers.py
You can also retrieve a stored function definition by slug when you want to inspect it locally:
bt functions pull --slug my-scorer --language python \
--output-dir ./pulled-functions
Keeping changes in Git helps you understand why scores move. If a coding agent makes the scorer more permissive while fixing the application, the higher score may reflect the new scoring rule. Reviewing that diff gives you a chance to catch the change before treating it as evidence that the application improved.
Give your coding agent a way to check its work
When your coding agent can run evals against the behavior you care about, you can ask it to check a proposed change before you accept it. Production logs give that work a connection to what your users experience, so a reported failure can inform the next fix and the test that checks it.
That's a practical reason to bring eval-driven development into your conversations with a coding agent. The CLI quickstart covers setup, and the command reference has the options for your workflow. If you're new to Braintrust, you can start free.
Also read: How to turn LLM production failures into regression tests and How to trace what your AI agent actually did
FAQs: using the Braintrust CLI
How do I install the Braintrust CLI?
With npm, run:
npm install -g @braintrust/bt
bt --version
The quickstart also documents standalone, pnpm, and mise installation options.
What's the difference between bt and the SDK?
You use the SDK to define your eval logic in code and instrument your application. The CLI gives you a way to run those evals from the terminal, along with commands for managing your Braintrust resources. They work together; see run evaluations for the available execution paths.
Can I run the Braintrust CLI in CI?
Yes, bt eval works on a macOS or Linux runner with the CLI and project dependencies installed. Supply credentials through CI secrets and use --no-input so the run does not wait for interactive input. The CLI quickstart has a GitHub Actions example.
How do I authenticate in scripts?
Set BRAINTRUST_API_KEY in the script's environment. For commands that need a project, pass --project explicitly or configure BRAINTRUST_DEFAULT_PROJECT. The CLI overview explains authentication and context selection.