Tutorial 4: RAG in Production

Contributor
4 days ago
2 min read

Naive RAG = chatbot demo. Production RAG = retrieval quality + safety + cost discipline.

Step 1: Naive RAG (10 min)

def rag(question):
    docs = vector_db.search(embed(question), k=5)
    context = "\n\n".join(d.content for d in docs)
    return llm.chat(f"Answer: {question}\n\nContext: {context}")

Works in demos. Mediocre in production.

Step 2: Retrieval Quality is Everything (15 min)

If retrieval misses relevant doc: LLM hallucinates.

Improvements:

Hybrid search (Path 78 Tutorial 7)
Reranking (Tutorial 7 of Path 78)
Better embeddings
Query rewriting
Multi-query retrieval

Spend 80% of RAG effort on retrieval. LLM is the easy part.

Step 3: Query Rewriting (15 min)

User: "what about pricing for the pro plan"

Bad embedding match (vague).

Rewrite:

rewritten = llm.chat(f"""
Original question: {question}
Conversation history: {history}

Rewrite as a standalone, specific question:
""")

Result: "What are the costs and features of the Pro tier?"

Better embedding match.

Step 4: Chunking Strategy (15 min)

How docs are split affects retrieval:

Fixed size: 500-1000 tokens with overlap
Recursive: by structure (paragraphs, sentences)
Semantic: by meaning shifts
Document-aware: per heading

For PDFs / docs: structure-aware better.

Too small chunks: missing context.

Too big: noisy retrieval; cost.

Step 5: Metadata Filters (15 min)

Don't search everything:

docs = vector_db.search(
  embed(query),
  k=5,
  filter={ 'product': 'pro', 'category': 'pricing' }
)

Identify intent → filter → search.

For multi-product / multi-tenant: critical.

Step 6: Citations (15 min)

Show sources:

prompt = f"""
Answer based on these sources. Cite source IDs in your answer.

Sources:
[1] {doc1.content}
[2] {doc2.content}
...

Question: {question}
"""

Output:

Pricing for Pro plan is $50/month [1]. It includes ... [2].

User can verify; builds trust.

Step 7: Handling Out-of-Scope (15 min)

User asks something not in your docs:

If the question isn't answered by the context, say:
"I don't have information about that. Please contact support."

Better: detect via heuristic (no relevant docs found) → don't even invoke LLM.

Hallucinations come from forcing answers to ungrounded questions.

Step 8: Updating Knowledge (15 min)

Docs change. RAG index needs updates.

Patterns:

Webhooks on doc update → reindex
Scheduled crawl
Per-customer indexes for SaaS

Tracking:

Doc → chunks → embeddings (lineage)
Updated_at on chunks

Without: stale answers; user trust lost.

Step 9: Conversation Memory (15 min)

Multi-turn:

def rag_chat(question, history):
    rewritten = rewrite(question, history)
    docs = retrieve(rewritten)
    answer = generate(question, docs, history)
    return answer

History context affects rewriting + answering.

Watch context window: truncate old turns.

Step 10: Production Checklist (15 min)

[ ] Hybrid retrieval (not pure vector)
[ ] Reranking
[ ] Query rewriting for conversations
[ ] Citations in output
[ ] Refusal for out-of-scope
[ ] Latency budget (< 5s ideal)
[ ] Cost tracking
[ ] Evals before deploy
[ ] Logging for investigation
[ ] Update pipeline

Each: hours-days of work. Worth it.

What You Just Did

RAG production: naive vs production, retrieval quality, query rewriting, chunking, metadata filters, citations, out-of-scope, updating, conversation memory, checklist.

Common Failure Modes

Naive RAG shipped to prod. Hallucination festival.

No citations. Trust loss.

No refusal mechanism. Always-answer creates lies.

Stale index. Wrong info served.

No evals. Drift unnoticed.

ShiftQuality