top of page

RAG Readiness Checklist

S. West
2 hours ago
4 min read

The HR assistant answered every demo question, then gave a manager the wrong notice period, with a citation.

The cited passage came from the contractor agreement, which sat next to the handbook in embedding space. The team tried a larger model first and got the same wrong answer in better prose; reading the retrieved chunks found the fault in minutes.

(Composite example, assembled from patterns common in internal LLM deployments. The figures are illustrative.)

This worksheet starts from your users' real questions, not the model. Retrieval-augmented generation is a design pattern, not a product (the term comes from Lewis et al., 2020), and its quality rests mostly on what gets retrieved. Rule it in or out, build the simplest loop, check five decisions, then let the diagnostic pick one component to fix.

Step 0: decide whether this should be RAG at all

Tick any that apply:

  • ☐ Answers need reasoning over the whole corpus, not a handful of documents (for example, "what themes ran through every support ticket this year?"). Consider aggregation or analytics.

  • ☐ Data changes faster than you index it (hourly edits, nightly embeddings). Fix the indexing cadence first; if the data truly changes per request, use direct queries or live APIs.

  • ☐ The task isn't retrieval (drafting an email needs no documents; working out tax on an order needs a function call). Route it to generation or a tool.

Write down the three questions users will ask most often:

Mark each as retrieval or not. If two or more are not, RAG is the wrong default and will make the system slower, dearer and less accurate.

Step 1: build the simplest loop, then record where it fails

  • ☐ The loop runs end to end: retrieve the most relevant passages, augment the prompt with them, generate an answer grounded in them. It is a few dozen lines of code and works surprisingly well on simple cases.

  • ☐ It was tested on the Step 0 questions, and failures were recorded per question before anyone chose a fix.

For each failing question, circle the closest symptom:

wrong documents · near-miss documents · right documents, answer ignored them · no citation · too slow or too costly · stale data

Step 2: chunking

Oversized chunks bury the answer in noise; undersized ones lose its context. No size suits every corpus, so iterate.

  • ☐ We checked for split context: a leave policy retrieved without the paragraph defining eligibility.

  • ☐ We compared retrieval quality across at least two chunk sizes.

  • ☐ Chunk boundaries follow document structure where it matters.

Chunk sizes tested, and what broke at each: ______________________________

Step 3: retrieval method

Embeddings find things that are similar, not things that are the same, so "employee termination" can surface "contract termination". Hybrid search (vector similarity plus keyword matching) often beats either alone, especially on terminology-heavy enterprise content. Test it on your own questions.

  • ☐ We know whether we run pure vector, pure keyword or hybrid retrieval.

  • ☐ We checked for near misses, and on specialised terms hybrid retrieval is in place or tested.

One near miss our retrieval returns, and whether keyword matching would have caught it: ______________________________

Step 4: re-ranking

A re-ranker is a second model that rescores retrieved passages against the query. It adds latency, and it often improves precision noticeably. Measure it on your data.

  • ☐ We measured a re-ranking pass on precision and weighed its latency cost.

Of our top retrieved passages, how many are relevant, and would re-ranking pay for its latency? ______________________________

Step 5: context curation

Passages, system prompt and question share one window, and more context costs latency and money. And LLMs don't always use all context equally: Liu et al. (2023) found models used information at the beginning or end of a long context more reliably than information in the middle. Newer models vary, so test yours.

  • ☐ We select context deliberately instead of loading every passage.

  • ☐ We handle the middle-of-context risk through ordering, re-ranking or summarisation.

  • ☐ Our design question is "how little does the model need to be right?", not "how much will fit?"

Fewest passages that still answer our top question correctly: ______________________________

Step 6: grounding and citation

A system that answers without showing its sources gives users no way to tell a grounded answer from a hallucinated one. Grounding means the model points to checkable sources, for example "the March travel policy, section 4, says…".

  • ☐ Answers cite sources a user can open and check.

  • ☐ Where errors are costly (compliance, clinical or legal work) we have considered citation verification: code that checks each claim against the retrieved text.

Does our domain need citation verification, and what does one confident wrong answer cost? ______________________________

The diagnostic: fix the failing component

Match the symptom to the component to examine first, not to a bigger model.

Symptom in production

Diagnose first

Wrong documents entirely

Chunking, then retrieval method

Near-miss documents

Retrieval method: add or tune hybrid search

Right documents, low precision in the prompt

Re-ranking

Answer ignores or drifts from the context

Context curation: ordering and minimum necessary

Confident answers with no checkable source

Grounding and citation

Slow, costly or stale

Step 0: is RAG the right pattern?

How to use it

  • Who: the engineers and the engineering lead who own the feature, in one sitting. Start with Step 0.

  • Output: three real questions, a symptom per failure, a filled prompt per step and one component to fix next.

  • Done when: every failing question maps to a component in the diagnostic, and the next sprint targets that component rather than the model.

Where this changes

Small teams with one document set can run Steps 0, 1 and 3, returning to the rest when a symptom appears. Regulated teams should require Step 6 citation verification and keep the filled worksheet as design evidence.

The reasoning is in RAG architecture for real data. For the next layer, see retrieval tuning and caching, costs and scaling.

Final takeaway

Most RAG failures are retrieval failures that look like model failures. Name the symptom, find its component, and fix that first.

Sources

bottom of page