Best agent skills for LLM evaluation

What are agent skills for LLM evaluation?

Each Braintrust skill is a readable SKILL.md file containing instructions that steer a coding agent through a specific eval task. Read the file before granting the agent access to a project. The instructions do not themselves read data or execute code. What the agent accesses depends on the task, its connected tools, and the permissions you grant. Without that guidance, an agent asked to "evaluate my chatbot" may run a few convenient examples and report a score without checking whether the cases represent real failures or the scorer rewards the right behavior.

Braintrust skills help your coding agent produce the datasets, scorers, and experiments behind an eval, so you can inspect the cases and decisions behind a score. Choose a skill for the task at hand rather than following every skill in sequence.

The planning, scorer-writing, validation, analysis, and gate-drafting instructions can work from local files or exports without a Braintrust account. A live Braintrust experiment requires an account, a project, and an API key. Reading hosted traces, saving datasets or scorers, and creating validation runs also require project access. Installing a skill does not grant any of that access. Review its SKILL.md and the agent's proposed reads, writes, and costs before authorizing a run; never paste an API key into chat.

Task-to-skill quick reference

Choose the skill for the job, then check what it needs before asking the agent to run it.

braintrust-plan-agent-eval

Does: Inventories the eval setup and identifies the earliest blocking gap.

Needs: An agent or product goal, not an existing dataset.

Produces: A dated plan naming the next artifact.

Reads/writes: Reads local project files and, if connected, Braintrust project objects; writes a plan, not a dataset or experiment.

A vague request like "evaluate this agent" can send a coding agent straight into writing an eval script. Use braintrust-plan-agent-eval when the next step is unclear. It inventories what exists and identifies the earliest blocking gap rather than assuming you need a new setup.

For example, ask, "Inspect my support agent's eval setup. Inventory the dataset, scorers, and experiments, then plan how to test whether its answers cite the correct policy." The skill produces an inventory of what exists and an ordered plan for what to create next. If the project has no dataset, the plan starts there. If it has a dataset but no scorer, the scorer becomes the next task. If both exist, the plan can move to an experiment.

You walk away with a specific next artifact plus a plan for reaching an inspectable Braintrust experiment.

With no eval setup yet, ask, "I have a new FAQ agent prototype and no traces, dataset, or scorer. Help me define what a correct answer looks like, identify what I must instrument, and name the first eval artifact to build. Do not create project resources yet." The plan can identify instrumentation or a behavior definition as the first gap, before you start collecting cases.

braintrust-eval

Does: Executes an eval with approval and smoke-run checks.

Needs: A runnable app or task, approved data and scorer criteria, a Braintrust project, and credentials.

Produces: A Braintrust experiment.

Reads/writes: Reads local code, dataset cases, and a Braintrust API key; imports or creates a dataset if needed and writes experiments and traces to the selected project after approval.

Use braintrust-eval when you want to execute an eval in a live project. The runbook can help import or build a dataset and write scorers, but it asks you to approve those choices and the full run before spending money. For example, ask your coding agent, "Run our support agent eval against the existing escalation dataset. Review the dataset and scorers, run a smoke test, then give me the Braintrust experiment link."

The agent checks the cases and scoring criteria, then runs a small smoke experiment behind its smoke gates and quota preflight. You review the smoke results before approving the full run, so a broken scorer or application error does not roll straight into it. Once the run finishes, you can inspect its scores and individual cases in Braintrust's experiment view. For traced runs, you can also examine the model calls and tool steps behind a failed case. The output is an experiment you can investigate and compare with later runs, rather than a result that lives only in the agent's console.

braintrust-build-eval-dataset

Does: Sources, labels, and audits eval cases.

Needs: Example cases, accessible traces, or a defined population and cases to create.

Produces: A documented, versioned dataset.

Reads/writes: Reads local cases or Braintrust traces if available; drafts cases locally and can write a dataset to a Braintrust project when authorized.

A passing eval suite can still miss the requests that fail in production. Use braintrust-build-eval-dataset when you have only happy-path examples, missing labels, or cases that no longer reflect real usage. The skill helps your coding agent source production cases, identify coverage gaps, and label expected behavior so a scorer has something meaningful to check.

Ask your agent, "Review our support-agent traces from the past 30 days. Build a Braintrust eval dataset that covers successful resolutions, incorrect tool use, and missed escalations. Remove sensitive information, label the expected behavior, and document how you selected the cases."

The deliverable is a documented, versioned dataset, not an unexplained collection of examples. Its documentation records where cases came from, what the labels mean, and which behaviors the cases cover. A version lets you compare experiments against a known set of cases as you add or revise examples.

braintrust-write-eval-scorer

Does: Turns one quality criterion into scoring logic.

Needs: A criterion and representative cases or outputs.

Produces: A versioned scorer and its failure rules.

Reads/writes: Reads local examples and code; can write a scorer locally, while saving it to Braintrust or running a judge requires project access and credentials.

"This answer looks good" is not a scoring rule an agent can apply consistently. Use braintrust-write-eval-scorer when you have eval cases but one quality requirement still depends on informal judgment. Ask the agent to define a passing output and build a scorer for that single criterion.

Choose a code-based scorer when the criterion has an objective rule, such as whether an answer contains a required field. For a judgment like factual support, ask for an LLM-as-a-judge rubric with examples that anchor both passing and failing scores. The scorer should specify how it handles missing reference material, empty outputs, and judge errors so those cases do not receive misleading scores.

For example, ask, "Write a scorer for factual support in our support answers. Use each dataset case's reference text, include examples of supported and unsupported claims, and mark cases with missing references as unscorable." You should get a scorer focused on that criterion, along with examples you can inspect before using it across the dataset.

braintrust-validate-eval-scorer

Does: Checks scorer agreement, false acceptances, and shortcuts.

Needs: A scorer and independently reviewed human labels.

Produces: A fitness verdict for exploration, monitoring, or release gating.

Reads/writes: Reads scorer outputs and human labels; can compare local files, while a Braintrust validation experiment reads a project and writes experiment results.

A scorer that runs without errors can still reward the wrong answers. Use braintrust-validate-eval-scorer before relying on its scores in CI or after changing its rubric. The skill compares scores with independent human labels and examines disagreements, especially false acceptances where the scorer passes an unacceptable output.

Ask your agent, "Validate our answer-grounding scorer against human labels. Find unsupported answers it passes, check whether citations let answers score well without supporting their claims, and tell me whether the scorer is fit for CI."

The skill produces a fitness verdict, including the errors and shortcuts you should address before relying on the scorer for a release decision. Adjudicated human labels are the reference tier its fitness bands are calibrated against, so the verdict is only as good as the labels behind it. Run it before adding a new scorer to a release gate, and run it again after changing the scorer's rules or rubric.

braintrust-analyze-eval-experiment

Does: Audits and compares completed eval results.

Needs: Completed runs, preferably on matching cases.

Produces: A comparison with uncertainty and subgroup findings.

Reads/writes: Reads exported results locally or Braintrust experiments with project access; writes analysis, not a new experiment by default.

A higher average score can hide a regression in one subgroup. Use braintrust-analyze-eval-experiment after a run finishes to find out whether the result supports a decision. The skill checks run completeness before comparing matching cases with a baseline, then reports uncertainty and subgroup findings.

For example, ask your agent, "Analyze the latest support-agent experiment against the baseline. Check for missing cases and scorer errors, compare matching examples, and tell me whether refund requests regressed." The analysis might find that the overall score rose while refund requests scored lower. It should identify the affected cases for inspection in Braintrust and say when too few paired cases support a conclusion.

braintrust-discover-agent-failures

Does: Finds unanticipated failures and freezes confirmed cases.

Needs: Agent runs with traces detailed enough to diagnose behavior.

Produces: A failure taxonomy and regression cases.

Reads/writes: Reads local trace exports or Braintrust traces; can write cases locally or to a versioned Braintrust dataset when authorized.

Some agent failures look like successes until you inspect the steps behind the answer. Use braintrust-discover-agent-failures when a passing eval suite does not match what you see in real runs. Ask your coding agent, "Review recent Braintrust traces for cases where the agent gave a plausible answer but ignored a tool result. Group the failures by root cause and propose regression tests." The skill examines traces for unexpected behavior, then organizes findings into a root-cause taxonomy so you can see which failure patterns recur.

The useful output is a set of durable test cases, not just a list of suspicious traces. For example, a trace where the agent answers from memory after receiving a contradictory tool result can become a labeled dataset case. You can use braintrust-build-eval-dataset to add that case to your eval and braintrust-write-eval-scorer to check whether future answers use the tool result correctly.

braintrust-define-eval-release-gate

Does: Defines enforceable ship-or-hold rules.

Needs: A held-out dataset, scorers validated for gating, and owned thresholds.

Produces: Human-readable guardrails and machine-readable CI rules.

Reads/writes: Reads local CI configuration and experiment evidence or exports; drafts rules locally, and changes CI or project gate settings only when authorized.

An eval dashboard can show a regression without blocking its release. Use braintrust-define-eval-release-gate once your dataset and scorers are fit for a ship-or-hold decision. The skill separates guardrail thresholds, which specify acceptable eval results, from CI rules, which specify when a run should block a release.

Ask your agent, "Define a release gate for our support agent. Require at least 95% correct policy citations, block any unsafe approvals, and fail CI if the evaluation does not complete." The resulting definition should identify the dataset and scorers, state each threshold, and specify how CI handles failed or incomplete runs. Thresholds that need a product owner's call are marked as needing a decision rather than guessed. You can then inspect the experiment behind a blocked release rather than relying on a single status value.

To wire the resulting rules into a pipeline, see how to run evals in CI.

Installing the skills

Install the Braintrust skills from the eval-library repository with the skills CLI.

npx skills add braintrustdata/eval-library

To install a specific skill, name it in the command.

npx skills add braintrustdata/eval-library --skill braintrust-eval

The CLI detects the coding agents installed on your machine and asks where to put the skills. Skills install into the current project by default; add -g to make them available globally.

The repository also documents a manual install if you would rather clone it and symlink the skill directories into ~/.claude/skills or ~/.codex/skills. Each skill directory is self-contained, so copying a single directory works too.

Skills follow the Agent Skills open standard, so they work in Claude Code, Cursor, Codex, and other compatible agents. The complete 24-skill catalog includes the eight skills covered here.

Picking the right starting skill

Starting from zero? Plan with braintrust-plan-agent-eval, build cases with braintrust-build-eval-dataset, write a criterion with braintrust-write-eval-scorer, and run the approved setup with braintrust-eval. Planning may first uncover a more basic gap, such as missing instrumentation or an unclear definition of good behavior.

Once you have results, use braintrust-validate-eval-scorer to check scoring against human labels, braintrust-analyze-eval-experiment to interpret completed runs, and braintrust-discover-agent-failures to find missed cases. Use braintrust-define-eval-release-gate only after the dataset and scorers are fit for a release decision.

Each skill produces an artifact you can read and challenge: a plan, a documented dataset, a scorer with its failure rules, an experiment, or a gate definition. That is the point of running evals through skills rather than ad hoc prompts — the score comes with the evidence behind it. Start free with Braintrust to inspect those artifacts in a project.