Retrieval that actually retrieves: a RAG checklist

When a RAG system gives a wrong answer, the instinct is to tune the prompt. In our experience, eight times out of ten the right passage never reached the model. Before touching the generation step, measure retrieval on its own: for a set of questions with known source documents, how often is the right chunk in the top five?

If that number is below 80%, no prompt will save you. Here is the checklist we work through.

Ingestion

• Is the text actually clean? PDFs exported to text often lose table structure, merge columns, or repeat headers on every page. Read twenty random chunks with your own eyes.
• Do chunks respect structure? Split on headings and paragraphs first, then by length. A chunk that starts mid-sentence and ends mid-table is hard to match and hard to use.
• Does each chunk carry context? Prepend the document title and section heading to every chunk before embedding. "Refunds are processed within 5 days" is much easier to retrieve when it also says Billing FAQ › Refunds.
• Are you deduplicating? Near-identical boilerplate (footers, disclaimers) crowds out the useful results.

Retrieval

• Hybrid beats pure vector. Combine keyword search (BM25) with embeddings. Product names, error codes and SKUs are exactly what embeddings blur together.
• Re-rank the top fifty. Retrieve generously, then use a cross-encoder or a small model to re-rank down to five. This is usually the single biggest quality jump.
• Filter before you search. If the user is asking about their own account, restrict to documents they can see before similarity search, not after.
• Rewrite the query. Conversational follow-ups ("what about for teams?") need to be rewritten into standalone questions before retrieval.

Generation

• Cite or abstain. Ask the model to quote the passage it used and to say "I don't know" when nothing relevant was retrieved. Then check that the quote really appears in the context.
• Keep the context small. Five excellent chunks beat twenty mediocre ones. Long contexts cost more and make the model more likely to latch onto the wrong passage.

Measurement

• Track recall@k separately from answer quality. When answer quality drops, you will immediately know which half of the system to look at.
• Log what was retrieved. Store chunk IDs with every answer. When a user reports a bad response, you can replay exactly what the model saw.