12 November 2026

Evaluating Amazon Bedrock Knowledge Bases

Jess Wang·12 min

Amazon Bedrock Knowledge Bases is AWS's managed RAG service that lets customers use business documentation and data to make AI smarter about their business. Ring uses it to create a customer-facing support chatbot for troubleshooting documentation. Miro uses it to enrich software bug reports with internal Jira tickets and source code.

The point of this eval is to measure how well Amazon Bedrock Knowledge Bases answers factual questions. How well does it retrieve usable evidence, generate a correct answer, include citations, and abstain when it does not have enough information?

Nova 2 Lite answered the most questions correctly and left the fewest unanswered. But when the expected source page was missing, it gave 28 of the 36 answers that weren’t supported by the sources it cited. Nova Lite v1 left the most questions unanswered, even when the retrieved text contained the answer.

Methodology

We used the KILT dataset, which is a frozen snapshot of Wikipedia from 2019, paired with questions and answers based on those pages. For each question, it also records the Wikipedia page that should contain the supporting information.

To run the evaluation, we ingested 100,000 pages from KILT's Wikipedia corpus into Amazon Bedrock Knowledge Bases. In this ingestion process, Bedrock:

1. Splits each Wikipedia source document into overlapping text chunks. We used 300-token chunks with 20% overlap.
2. Uses Amazon Titan Text Embeddings V2 to turn every chunk into a numerical embedding.
3. Stores each embedding in an Amazon S3 Vectors index.

We then selected a fixed set of 100 answerable KILT questions and verified that each question’s expected Wikipedia source page was among the 100,000 pages we ingested. So, when a user asks a question like "is a bee an animal or an insect?", this is what happens:

1. Bedrock embeds the question using the same embedding model.
2. It searches the vector index for the k chunks most semantically similar to the question.
3. RetrieveAndGenerate gives the retrieved k chunks to the selected generation model.
4. The model generates an answer based on the context.

Once the answer is generated, we score it against these main criteria:

1. Factuality: Did the answer correctly answer the question?
2. Claim support: What fraction of the answer's factual statements are directly supported by its cited evidence?
3. Gold-page recall: Did any of the Retrieve chunks come from the expected Wikipedia source page?
4. Supporting-chunk recall: Did any of the Retrieve chunks contain enough information to answer the question?
5. Generation response status: Did the system answer, abstain, produce bad output with or without a usable final answer, or hit a request error?[1]

We used Amazon Nova Lite v1, Amazon Nova 2 Lite, and Claude Sonnet 4.6 for our generation models. We chose these models because they were available through Bedrock. Accessing other models required additional permissions and sales approval.

GPT-OSS-120B was the primary LLM judge for factuality and claim support because it was cost-effective, and we wanted to avoid having the same models generate and judge their own output. For each generation model, we tested two retrieval depths, k=5 and k=10, and two answer styles. Natural allowed a normal-length response, while concise required the shortest direct answer.[2]

Analysis

Model performance

Task factuality measures accuracy across all questions, including those the model declined to answer. Answer-conditional factuality measures accuracy only among questions with a usable answer.

• Nova Lite v1 is the least robust model. Its task factuality[3] is 45.5%, compared with 64.0% for Nova 2 Lite and 60.5% for Sonnet.[4] It abstains on 37.3% of questions on average, versus 10.5% for Nova 2 Lite and 12.8% for Sonnet. In a production RAG pipeline, that means it will leave a large share of answerable questions unanswered.
• Nova Lite v1 has the highest answer-conditional factuality at 76.2%. This means that although the model abstains more, it is more correct for the questions it decides to answer.
• Nova 2 Lite has the highest observed task factuality. It scores 62–66% task factuality, abstains on only 8–12% of questions, and includes citations with source text in 89–97% of responses. For a production pipeline, it is the strongest starting model, though we will evaluate its robustness later in the blog.

With the corpus, embeddings, and chunking held fixed,[5] factuality, claim support, inclusion of citations, and abstention behavior still varied substantially across models.

Judge agreement

We also ran a small additional check on how well our judge was performing. On a sample of 120 responses, half marked correct and half incorrect[6], we swapped in DeepSeek R1 as a judge and found 86.7% agreement with our original judge, with Cohen's κ of 0.733.

DeepSeek was slightly more permissive. Of the 16 disagreements, it gave credit to 14 answers that GPT-OSS marked incorrect, while GPT-OSS gave credit to two answers that DeepSeek marked incorrect.

Retrieval depth and factuality

Retrieving more chunks did not reliably improve the main factuality result.

In the paired comparison, 33 answers to the same questions under the same model and answer style changed from incorrect at k=5 to correct at k=10. Twelve of those 33 improvements, or 36%, gained a new answer-supporting chunk at ranks 6 through 10.

However, these 12 improvements came from two repeated questions across the six configurations, so this is a very narrow benefit.

There were also 31 regressions, meaning an answer that was correct at k=5 is now incorrect at k=10. The top five standalone Retrieve chunks were identical for all 31 regressions, and 27 still had an answer-supporting chunk at both depths. In these cases, the model may choose a different answer, decide to abstain, or include different unsupported details after it sees the additional ranks 6 through 10.[5]

Abstention behavior

• Across the 1,200 responses, 944 returned valid answers, 242 abstained, 2 had formatting errors but a usable answer, and 12 had formatting errors with no usable answer.
• For 166 of 242 abstentions, no top-k chunk was judged sufficient to answer the question by the supporting-chunk recall scorer.[7] In those cases, abstaining makes sense given the retrieved text. The expected source page was ingested for every question, so the pattern suggests a retrieval or chunk-ranking miss rather than evidence that the answer was absent from the corpus.
• For the other 76 abstentions, the separate search returned an answer-supporting chunk. Nova Lite v1 accounts for 73 of those 76 cases, or 96.1%. Nova 2 Lite accounts for one and Sonnet for two. Nova Lite v1 also made 149 of all 242 abstentions, so abstentions where evidence is available are overwhelmingly a Nova Lite v1 response pattern.
• Forty-five of 242 abstentions (18.6%) cited at least one chunk even though the system abstained. Forty-one of those 45 had no answer-supporting chunk in the returned k chunks, so the citations were often related context used to explain the abstention. For example, one case cited an article introduction that named the relevant entity without containing the needed fact.

Concise versus natural answers

Averages across all 100 questions mix answer style with whether the model answered at all. This chart includes only questions where both concise and natural produced valid answers, so it compares answer style rather than abstention behavior.[8]

In this graph, each bar represents natural minus concise in percentage points. Positive values favor natural, and negative values favor concise.

Concise factuality was equal or higher in all six matched comparisons. A plausible explanation for the factuality pattern is that a shorter direct answer gives the model fewer chances to add an incorrect or irrelevant factual claim.

Model performance by question type

We grouped questions by type to identify patterns in what the system struggled to answer. One question can belong to more than one category.

• Multi-part answer: Requires multiple entities, like “who were the 3 presidents that died on july 4th.”
• Comparing or combining facts: Requires comparing entities or combining facts, like “real madrid or barcelona who has more trophies.”
• Role / relation: Requires precise knowledge of a relationship, like “who played dr. loveless in the wild wild west.”
• Ambiguous subject: The wording could refer to several people, like “who wrote something in the way she moves.”
• List / table evidence: The answer is likely buried in a list or table, like “how many episodes of the secret circle are there.”
• Time-sensitive answer: The answer depends on time, like “when is the next maze runner film coming out.”
• Fact near the start of the page: This group tests relatively easy retrieval, like “is a bee an animal or an insect.”

• Facts near the start of the page are easiest on average. They have the highest answer-conditional factuality at 65.3%, the strongest answer-supporting retrieval at 89.7%, and the lowest abstention rate at 11.7%. The answer is often near the beginning of a source, so retrieval is more likely to surface a self-contained answer. For a production pipeline, this is the baseline case. Failures here are a signal to inspect ingestion or retrieval before tuning generation.
• Comparison and multi-part questions are the hardest types of questions to answer. Comparison has only 54.5% supporting-chunk recall and 44.7% answer-conditional factuality. Multi-part answers have 36.7% answer-conditional factuality. Comparisons often require retrieving evidence about more than one entity, while multi-part questions require combining several facts into one complete answer. A production pipeline may need query decomposition, separate retrieval per entity, or explicit completeness checks for these cases.
• Roles and relationships are another group of questions the models struggle with. Their answer-conditional factuality is 48.5%, despite 69.3% supporting-chunk recall. This gap points to a mix of retrieval difficulty and model interpretation difficulty. Supporting evidence was often present, but the system still answered incorrectly, suggesting that the model struggles to identify or use the precise relationship the question asks about. These questions are good candidates for stricter extraction prompts or a verification step.

Another way to assess question difficulty is to compare the questions that all models struggled to answer with the questions that all models answered correctly.

Twenty-five questions, or 25% of the set, received zero answers that GPT-OSS judged factually correct in all four configurations of each model. In other words, every attempt for those questions either abstained, produced unusable output, or gave an answer GPT-OSS judged incorrect. At the other extreme, 20 questions, or 20% of the set, were correct in all four configurations of every model. These two groups help separate questions that are likely difficult across configurations from questions that are consistently easy, while the remaining 55 questions show where model and answer style change the outcome.

Abstention when the expected source page is missing

Nova 2 Lite concise at k=10 had the highest observed factuality on the 100 answerable questions. That is useful, but it is only one side of the decision. This experiment design rewards a system for answering when it has the required source. We also wanted to test whether the system could recognize when required evidence is missing and correctly abstain instead of hallucinating.

To test that behavior, we ran a separate 50-question negative control. For every question, we verified that its expected Wikipedia source page was excluded from the ingested corpus. Abstention is the conservative default outcome, although duplicate evidence can still appear on another ingested page.

Of the 600 responses, 506 (84.3%) abstained and 94 (15.7%) had an answer. Of the 94 answers, 58 were supported, meaning that although their expected Wikipedia source page was not in the corpus, other ingested articles contained enough evidence to answer the question. The other 36 (38.3%) were unsupported.[9]

Those 36 unsupported answers are 6.0% of all 600 responses in the control test. They are the strongest candidates for a failure to abstain because their cited sources contained no supporting evidence. Of these 36, 28 (77.8%) were from Nova 2 Lite. On the answerable set, Nova 2 Lite's willingness to answer contributed to its strong factuality and low abstention rate. In this control test, the same tendency produced most of the unsupported answers.

Considerations for deployment

Retrieval and generation can fail separately. We evaluated the generation portion of the pipeline, but a further iteration could test retrieval by varying chunk sizes and overlap settings, the embedding model, or by using hybrid search instead of semantic search.

It's also important to test questions whose expected source page was missing. Other pages may still contain enough evidence to answer, so check whether the cited sources support the response.

How readily the system should answer depends on the product you're building. An internal research assistant for analysts may reasonably answer more aggressively, as long as it shows citations and a person can verify them. A medical-triage tool should lean toward abstention when the evidence is insufficient. In those settings, a confident unsupported answer can have bad consequences.

If you want to evaluate a RAG system like AWS Bedrock Knowledge Bases, try Braintrust for free.