What's new
Explainer14 Nov 20265 min read

Find out whether your RAG system found the answer first

In our worked example, a document assistant answers 68 of 100 questions correctly, and twenty of the failures have no supporting passage in the final prompt.

When a document assistant gives a wrong answer, the tempting fixes are a larger model, a reworded question or more retrieved text. Look first at what the model was actually given. If the supporting passage never reached the prompt, an answer-quality score alone cannot say which fix to try.

Anthropic's retrieval explanation, read on 14 November 2026, describes a pipeline that finds document chunks and passes them to the answering model. That gives two things to check separately: whether the right material arrived, and whether the answer used it correctly. The numbers below are invented to show the split; they are not Anthropic's results.

Each question needs a known supporting passage

Pick questions whose answers you can establish from the collection, and save the source version and the passage each one needs. Where a question needs several passages, note that, because finding one half of a comparison is not complete evidence.

Our fictional test has 100 answerable questions. Evidence reaches the final prompt for 80, and 68 of those get an answer that passes review. Passing here means the answer is supported by the supplied material, so a lucky unsupported guess fails.

Synthetic nested outcomes for 100 answerable questions. Retrieved means the required supporting passage is present in the final generation prompt.
Gate in the invented runPassing questionsShare of all questions
Evidence available8080%
Answer accepted6868%

Retrieval coverage is 80% and overall acceptance is 68%. Among the cases where evidence was available, acceptance is 85%. All three figures belong in the report, since quoting only the last would hide that a fifth of the original questions never had their evidence.

Retrieval counts at the final prompt

A passage can show up in search results and vanish during reranking, deduplication or prompt assembly. So measure retrieval at the point that matters, the material actually supplied for generation, and keep the intermediate results privately for finding where a passage was lost.

For a failed case, first ask whether the document was in the indexed collection at all. Then look at the candidates, the selected passages and the assembled prompt in turn. That separates an ingestion gap from a search failure or an over-tight context budget. These are diagnostic steps we suggest, not measured causes from a real deployment.

Source identifiers should survive a rebuild. A chunk's position in an array can shift when another file is added, so store a document version and a location within it, which lets a reviewer find the evidence again.

Changing one thing at a time keeps the old cases useful

If evidence is missing, compare retrieval settings on the same questions before touching the answering model. If it is present but the answer fails, look at how the question and passages are presented. Save the full configuration for each run, or an improvement cannot be traced to the change you meant to test.

When your collection contains exact identifiers, test those as well as ordinary-language questions. Anthropic's article discusses combining lexical and embedding retrieval. That is an approach worth testing, not proof that a hybrid setup improves your collection, and any added delay and prompt tokens should sit beside the retrieval result.

Questions with no answer in the collection form their own category, testing whether the assistant admits the gap. Mixing them into a denominator defined as answerable would blur retrieval coverage, and for them an empty result can be correct.

What the results log needs to hold

The log should hold the collection version, indexing settings, question-set version, evidence-available count and accepted-answer count. Add the number of questions needing multiple sources and the failures among them. The test should stay small enough to inspect by hand before it grows.

A higher end-to-end score is welcome. A record of where the evidence was lost is what says what to try next, and whether the next model release is relevant to the problem you actually have.