How to test your RAG pipeline (before and after you ship)

Before you start testing your RAG pipeline

This runbook assumes a working RAG pipeline. If you need background on why retrieval and generation require separate evaluation, read what RAG evaluation is. Before you begin, make sure the following three requirements are in place.

Separate spans for retrieval and generation: Record retrieval and generation as distinct spans within the same trace, with the retrieved documents captured in the retrieval span's output. Braintrust supports span-level and trace-level scoring. Separate spans let you evaluate each stage on its own and narrow a regression to the stage that produced it.

Access to production query logs: Team-written synthetic questions can miss the phrasing, typos, ambiguity, and edge cases present in user traffic. Production logs provide representative queries to build and expand the test dataset.

A way to pin the configuration under test: Every evaluation run should record the corpus or index version, document-processing settings, embedding model, retrieval parameters, reranker configuration, generation model, and prompt version. Without this metadata, you cannot determine which pipeline change caused a score to move.

Step 1: Build a golden dataset from real RAG queries

A golden dataset provides a consistent set of test cases for measuring future pipeline changes. Begin with the queries that represent core user needs, then expand the dataset as experiments and production traffic expose new failure modes.

Choose a schema that supports each test

Braintrust datasets support input, expected, metadata, and tags. The input field should contain the query and any data needed to recreate the case. Depending on the scorer, expected can contain relevant document identifiers, a reference answer, or assertions. Metadata and tags can record the query type, source, and review status.

Not every record needs both expected documents and a complete reference answer. Label-based retrieval metrics require relevant document identifiers, while reference-based generation scorers require an expected answer. Assertion-based and reference-free scorers can evaluate cases without a full answer label.

Source test cases from production logs

Loop can review production logs and suggest dataset rows, so representative queries move into the golden dataset without manual filtering.

Production queries reveal the phrasing, ambiguity, and failure patterns users encounter. Once the RAG application is instrumented, you can review Braintrust traces and add selected cases to a dataset.

Braintrust recommends starting with 5 to 10 representative examples that cover core use cases. Add confirmed failures from experiments and production as the pipeline encounters new query patterns.

Label only what each test requires

For label-based retrieval tests, record the document identifiers that should appear in the results. You only need relevance labels for the selected queries, although the labels should be reviewed when the corpus changes.

Where complete reference answers are expensive to produce, assertions can define the requirements for an acceptable response. The Braintrust ToolRAG recipe uses assertions to test specific answer characteristics without prescribing exact wording. For example, an authentication question could require the correct header name and a working code example.

Keep the cases used for tuning separate from a holdout set reserved for release checks. The holdout set gives you an independent test of whether a change generalizes beyond the examples used during iteration.

Cover different query types

Include deliberate coverage across the following categories:

• Factual questions with one clear answer
• Multi-document questions that require information from several sources
• Ambiguous queries with multiple valid interpretations
• Unanswerable questions that require the system to acknowledge missing information

Record the query type and iteration or holdout status in metadata. Braintrust tracks dataset versions, so an experiment can pin the version it ran against while results remain filterable by query category.

Step 2: Test retrieval quality on its own

Evaluate the retrieved results independently of the generated answer so an end-to-end regression can be traced to document selection, ranking, or generation.

Score returned documents against relevance labels

Attach code-based scorers to the retrieval span and compare the returned document identifiers with the relevance labels recorded in Step 1. Recall@k measures how many known relevant documents appear among the top-k results. Precision@k measures how many of those results are relevant. Use NDCG when relevance is graded or rank order affects which passages reach the generation model.

Use context scorers for semantic retrieval checks

The Braintrust Autoevals library also provides LLM-based scorers for evaluating retrieved text. ContextRecall measures how well the context supports an expected answer, ContextPrecision assesses whether useful passages rank ahead of irrelevant ones, and ContextRelevancy evaluates how closely the context aligns with the query.

These scorers do not compute document-ID recall or precision, so select the appropriate metric based on the available labels and the retrieval failure being tested. Braintrust's RAG evaluation metrics guide covers both label-based and semantic measurements.

Establish a retrieval baseline

Run the current retrieval configuration against the dataset and save the experiment as the baseline. Record the top-k value, corpus or index version, embedding model, retrieval parameters, and reranker configuration so later score changes can be attributed to a specific revision.

Change one variable in each subsequent experiment, then compare both aggregate metrics and individual cases. Increasing top-k may improve recall, reduce precision, or leave either metric unchanged, so check the actual scores before approving the revised configuration.

Step 3: Test generation quality against fixed context

Even when the retriever returns the correct documents, the generation model can produce an incomplete, irrelevant, or unsupported response. Testing against reviewed, fixed context isolates changes caused by the prompt or generation model.

Hold retrieved context constant

Provide the same reviewed documents for each query across every experiment. With retrieval variation removed, differences in answer quality can be attributed to the generation configuration and normal model variability.

Score faithfulness, answer relevancy, and correctness

Braintrust provides three relevant generation scorers. Faithfulness checks whether the answer is supported by the supplied context. AnswerRelevancy measures how directly the response addresses the query, while AnswerCorrectness compares the response with an expected answer when one is available.

Review faithfulness and relevancy independently because a response can address the query while introducing claims that the context does not support. Use correctness when the dataset contains suitable reference answers.

Validate the scorer before using it for release decisions

In the recipe's recorded run, changing only the grading model moved AnswerCorrectness from 67.28% to 72.10% while ContextRecall stayed at 95.00%, so the difference comes from the judge, not the pipeline.

Compare each LLM judge with human-reviewed examples and inspect disagreements before allowing its scores to influence a release. Braintrust's Ragas evaluation recipe demonstrates how changes to the grading model can affect the reported results.

The recipe runs the same evaluation twice, changing only the model that grades AnswerCorrectness. Its first experiment recorded an AnswerCorrectness score of 67.28%. The second passes a different grading model to the scorer while leaving the RAG task and the ContextRecall scorer untouched:

# Wrap ContextRecall() to propagate the "answer" and "context" values separately
async def context_recall(output, **kwargs):
    return await ContextRecall().eval_async(output=output["answer"], context=output["retrieved_docs"], **kwargs)


# The grading model is the only variable that changes between the two experiments
async def answer_correctness(output, **kwargs):
    return await AnswerCorrectness(model=GRADING_MODEL).eval_async(output=output["answer"], **kwargs)


eval_result = await EvalAsync(
    name="Rag Metrics with Ragas",
    experiment_name=f"Score with {GRADING_MODEL}",
    data=qa_pairs[:NUM_SECTIONS],
    task=generate_answer_e2e,
    scores=[context_recall, answer_correctness],
    metadata=dict(model=QA_ANSWER_MODEL, topk=TOP_K),
)

The recipe's second experiment raised AnswerCorrectness to 72.10%, with 10 cases improving and 4 regressing, while ContextRecall remained at 95.00%. The specific models matter less than the pattern: naming the experiment after the grading model keeps the two runs distinguishable, and the scores move even though the pipeline never changed. Inspecting those case-level differences helps determine whether the revised judge is more accurate or applies a different grading standard.

Step 4: Test the complete pipeline and score a change before shipping

Testing retrieval and generation separately identifies where a failure begins. Before release, run both components together to confirm that the complete RAG pipeline produces the required result when retrieval, context assembly, and generation interact.

Run an end-to-end experiment

Comparing a candidate experiment against the baseline shows which individual cases improved or regressed, with the retrieval and generation spans available for inspection.

Execute the normal pipeline against the dataset from Step 1. Keep retrieval and generation scorers attached to their respective spans, then add trace-level scorers or deterministic assertions for the final outcome. These checks can verify that citations resolve to retrieved sources, required information appears in the answer, and unanswerable queries are handled correctly.

Run the current configuration first, then evaluate the proposed change against the same dataset snapshot and scorer versions. Braintrust experiments preserve each run, so you can compare new results against the baseline at both the aggregate score and the individual test case level.

Change one variable per experiment

Change one configuration variable between the baseline and candidate experiment. Updating the chunking strategy and the generation prompt in the same run would make it impossible to attribute the resulting movement in the score. If a release contains several changes, evaluate them separately before testing the combined configuration.

When identical inputs can produce different outputs, use trial_count to run each case multiple times. Repeated trials provide an average score and expose cases where the proposed configuration performs inconsistently.

Review improvements and regressions by test case

The Braintrust Ragas recipe shows why the aggregate score cannot determine a release on its own. Reducing top-k from two documents to one moved AnswerCorrectness from 72.10% to 71.99%, a decline of only 0.12 percentage points. However, 9 cases improved and 11 regressed.

Inspect each regression to determine which queries were affected and whether the retrieved evidence or generated answer changed. A regression on an authentication, billing, or other critical query may outweigh several improvements on lower-risk cases.

Define the release requirements in advance

Set the acceptance criteria before reviewing the candidate experiment. Specify the minimum result for each scorer, the permitted number of regressions among previously passing cases, and any cases that must always pass. Unanswerable queries should have a separate requirement that stops unsupported answers from passing.

Each condition should identify the scorer, threshold, and applicable test group so the same requirements can be enforced in CI.

Step 5: Gate RAG changes in CI

Running evaluations manually leaves release requirements dependent on someone remembering to run them. Adding the RAG test suite to continuous integration enforces the requirements defined in Step 4 before a change reaches the main branch.

Run evals on every pull request

The Braintrust GitHub Action runs evaluations when a pull request targets main and posts the results as a comment.

name: Run evaluations

on:
  pull_request:
    branches: [main]

permissions:
  pull-requests: write
  contents: read

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 24
      - name: Install dependencies
        run: npm install
      - name: Run evals
        uses: braintrustdata/eval-action@v2
        with:
          api_key: ${{ secrets.BRAINTRUST_API_KEY }}
          runtime: node

Set runtime to node, python, or go based on the project. The pull-requests: write permission allows the action to post its results as a pull request comment.

Run a smoke test on pull requests and the full suite on merge

LLM-based scoring can make the complete RAG test suite too slow or costly for every pull request. The bt eval CLI supports a limited, non-final run for pull requests and a complete run after merge.

bt eval tests/ --first 20 --no-input --json # smoke run on PR, non-final
bt eval tests/ --no-input --json # full run on merge, final

The pull request subset should include critical queries and representative failure cases. The complete run then verifies the release against the full dataset before the revised pipeline is deployed.

Define pass and fail with a custom reporter

By default, bt eval returns a non-zero exit code when an evaluation throws an exception. A lower quality score does not automatically fail the command. Custom reporters return a non-zero exit code when the release requirements are not met.

from braintrust import Reporter


def report_eval(evaluator, result, opts):
    # Summarize the results of a single evaluator, and return whatever you
    # want (the full results, a piece of text, or both!)
    pass


def report_run(results):
    # Take all the results and summarize them. Return a true or false
    # which tells the process to exit.
    return True


Reporter(
    "My reporter",  # Replace with your reporter name
    report_eval=report_eval,
    report_run=report_run,
)

Replace return True with logic that evaluates the scorer thresholds, permitted regressions, and must-pass cases defined in Step 4. Returning False causes bt eval to exit with a non-zero status, blocking the merge in the CI workflow.

Step 6: Score production traffic after release

Offline experiments evaluate the cases already captured in the dataset. After release, new query patterns and corpus changes can expose failures that the pre-release suite does not cover. Production scoring detects those cases while the request context is still available for diagnosis.

Configure online scoring for production traces

Online scoring attaches scores and rationales to production traces asynchronously, so a failed answer can be diagnosed with its retrieval spans still available.

Braintrust online scoring evaluates traces asynchronously, so scoring does not add latency to the user-facing request. Each project-level rule defines the scorers, trace or span scope, log filters, and sampling rate.

A scorer used in offline experiments can run in production when the required inputs are present in the trace. Many live requests lack an immediate reference answer, so Faithfulness, AnswerRelevancy, and ContextRelevancy are more broadly applicable than expected-answer scorers. User feedback, confirmed outcomes, and changes in the business state can serve as ground truth when captured by the application.

Choose the sampling rate based on traffic volume, request risk, and scoring cost. Use log filters to score critical flows more heavily and sample routine high-volume traffic.

Measure retrieval relevance in production

A span-level context scorer can compare the retrieved text with the query without requiring known relevant document identifiers. The resulting score measures semantic relevance or usefulness, not conventional Precision@k, which requires relevance labels or another confirmed signal.

Record the query category, corpus or index version, and retrieval configuration in log metadata. Filtering scores by those fields can reveal whether a decline is concentrated in a specific query type or began after a corpus update.

Track score movement and alert on failures

Braintrust dashboards can track aggregate scores and distributions over time. Log alerts use SQL filters to notify the team when an individual production log meets defined conditions, such as a faithfulness score falling below the accepted threshold for a critical query category.

Because online scores are attached asynchronously, the alert filter should confirm that the relevant score exists before testing its value. Use dashboards to investigate sustained aggregate movement and log alerts to surface individual failures that require attention.

Add confirmed failures to the dataset

Filter production logs by score and metadata, then review the lowest-scoring traces before adding them to the dataset. Confirmed failures should receive the relevant document labels, expected answer, or assertions needed for future evaluation. Braintrust promotes traces from logs into datasets, so observed production behavior expands the regression suite.

Braintrust matches identical cases across experiments, even when the dataset version changes. However, newly added cases have no result in the earlier baseline, and aggregate scores may cover different case sets. Rerun the current production configuration against the updated dataset version before evaluating the next pipeline change.

RAG testing checklist: before and after you ship

Before you ship

• Retrieval and generation are recorded as separate spans
• Golden dataset includes the labels or assertions required by each scorer
• Factual, multi-document, ambiguous, and unanswerable queries are represented and tagged
• Retrieval is evaluated independently of the generated answer
• Generation is tested against reviewed, fixed context
• LLM judges are validated against human-reviewed cases
• The complete pipeline is tested against a saved baseline
• Configuration versions are recorded in experiment metadata
• Aggregate results and case-level regressions are reviewed
• Release requirements are defined before candidate results are reviewed
• CI runs a representative subset on pull requests and the complete suite before deployment

After you ship

• Online scoring rules specify the scorer scope, log filters, and sampling rate
• Reference-free scorers evaluate production traces without adding request latency
• Retrieval relevance is tracked by query type and configuration version
• Dashboards track aggregate score movement over time
• Log alerts surface individual failures that cross defined thresholds
• Low-scoring traces are reviewed before confirmed failures enter the dataset
• The current production configuration is rerun after a material dataset update

Run your RAG tests in Braintrust

The Playground runs two RAG configurations against the same dataset rows and scores each output, so a candidate change can be compared before a full experiment.

Braintrust connects production traces with versioned datasets, experiments, and CI. Once a production failure is confirmed, you can add it to a dataset, evaluate the next pipeline revision against a saved baseline, and determine whether the correction meets the release requirements.

Scorer thresholds, permitted regressions, and must-pass cases can then be enforced through CI, while product, support, and engineering teams can contribute test cases and review differences in experiments within Braintrust. Loop, Braintrust's AI agent, lets a support lead or PM analyze logs, find related traces, and build datasets or scorers using natural-language instructions, without having to file a ticket with engineering.

Dropbox uses Braintrust for Dash, its AI-powered search product. The team moved from spreadsheets and ad hoc coordination to an evaluation pipeline that runs more than 10,000 tests and detects regressions in real time.

Braintrust's free plan includes 1 GB of processed data, 10,000 scores per month, and unlimited users, projects, datasets, playgrounds, and experiments.

Start testing your RAG pipeline with Braintrust.

FAQs: How to test a RAG pipeline in 2026

How do I test my RAG pipeline?

Start by defining a successful result for each user task. A support answer may need to cite the correct policy, address the customer's question, and acknowledge when the available documents lack sufficient information. Evaluate each requirement separately, then compare the proposed pipeline version with the version currently approved for production.

How do I know a change improved accuracy before I ship it?

Run the current and proposed versions under the same conditions, including the dataset, corpus, scorers, and number of trials. An improvement is credible when it remains consistent across repeated runs and relevant query categories without reducing quality on critical cases. Braintrust experiment comparisons show case-level changes that an aggregate score can conceal.

How do I measure RAG retrieval precision in production?

Precision@k requires relevance judgments for the retrieved records. Sample production queries, label each of the top-k results as relevant or irrelevant, and calculate the proportion of relevant records for each query. Report the results by query category because a single overall figure can hide weak retrieval for less common or higher-risk requests.

What tools run automated RAG evals?

Braintrust runs automated RAG evaluations across development and production. You can execute saved test cases as experiments, rerun them in CI, and apply configured scorers to sampled production traces after deployment. Using consistent evaluation criteria across each stage connects release approval with ongoing quality measurement.

How do I track the correctness of RAG answers over time?

Maintain a fixed benchmark alongside a rolling sample of current production queries. The benchmark reveals regressions against established behavior. The production sample catches changes in user requests and source content. Record the scorer, model, prompt, and corpus versions with every result, and create a new baseline whenever the grading configuration changes.

How large should my RAG golden dataset be?

Dataset size should reflect query coverage, release risk, and the smallest quality change the evaluation needs to detect. Hundreds of similar questions can still provide weak coverage when critical intents have only one or two cases. If small score differences will influence release decisions, use the observed score variance to estimate how many cases or repeated trials are needed for a reliable comparison.

Can I test RAG without ground truth answers?

A complete reference answer is unnecessary when acceptable behavior can be expressed through evidence and specific requirements. For example, a response can be checked for source-backed citations, required policy details, and an appropriate acknowledgment when the retrieved context is insufficient. Reference-free judges should be calibrated against human-reviewed cases and audited periodically so changes in the judge are not mistaken for changes in RAG quality.