top of page

RAG in Production — LLM Application Engineering, Part 4

  • Shawn West
  • Jul 8
  • 3 min read

Updated: Aug 6

LLM Application Engineering · Part 4

Naive RAG — embed the docs, stuff the top matches into the prompt — makes a great demo and a disappointing product, because in production the answer is only ever as good as what retrieval hands the model. Feed it the wrong three chunks and it will confidently cite them anyway. This walks through the gap between demo RAG and production RAG: fixing retrieval quality, chunking deliberately, adding citations and out-of-scope handling, and keeping the knowledge base fresh.

Naive RAG = chatbot demo. Production RAG = retrieval quality + safety + cost discipline.

Step 1: Naive RAG (10 min)

def rag(question):
    docs = vector_db.search(embed(question), k=5)
    context = "\n\n".join(d.content for d in docs)
    return llm.chat(f"Answer: {question}\n\nContext: {context}")

Works in demos. Mediocre in production.

Step 2: Retrieval Quality is Everything (15 min)

If retrieval misses relevant doc: LLM hallucinates.

Improvements:

  • Hybrid search (Path 78 Part 7)

  • Reranking (Part 7 of Path 78)

  • Better embeddings

  • Query rewriting

  • Multi-query retrieval

Spend 80% of RAG effort on retrieval. LLM is the easy part.

Step 3: Query Rewriting (15 min)

User: "what about pricing for the pro plan"

Bad embedding match (vague).

Rewrite:

rewritten = llm.chat(f"""
Original question: {question}
Conversation history: {history}

Rewrite as a standalone, specific question:
""")

Result: "What are the costs and features of the Pro tier?"

Better embedding match.

Step 4: Chunking Strategy (15 min)

How docs are split affects retrieval:

  • Fixed size: 500-1000 tokens with overlap

  • Recursive: by structure (paragraphs, sentences)

  • Semantic: by meaning shifts

  • Document-aware: per heading

For PDFs / docs: structure-aware better.

Too small chunks: missing context.

Too big: noisy retrieval; cost.

Step 5: Metadata Filters (15 min)

Don't search everything:

docs = vector_db.search(
  embed(query),
  k=5,
  filter={ 'product': 'pro', 'category': 'pricing' }
)

Identify intent → filter → search.

For multi-product / multi-tenant: critical.

Step 6: Citations (15 min)

Show sources:

prompt = f"""
Answer based on these sources. Cite source IDs in your answer.

Sources:
[1] {doc1.content}
[2] {doc2.content}
...

Question: {question}
"""

Output:

Pricing for Pro plan is $50/month [1]. It includes ... [2].

User can verify; builds trust.

Step 7: Handling Out-of-Scope (15 min)

User asks something not in your docs:

If the question isn't answered by the context, say:
"I don't have information about that. Please contact support."

Better: detect via heuristic (no relevant docs found) → don't even invoke LLM.

Hallucinations come from forcing answers to ungrounded questions.

Step 8: Updating Knowledge (15 min)

Docs change. RAG index needs updates.

Patterns:

  • Webhooks on doc update → reindex

  • Scheduled crawl

  • Per-customer indexes for SaaS

Tracking:

  • Doc → chunks → embeddings (lineage)

  • Updated_at on chunks

Without: stale answers; user trust lost.

Step 9: Conversation Memory (15 min)

Multi-turn:

def rag_chat(question, history):
    rewritten = rewrite(question, history)
    docs = retrieve(rewritten)
    answer = generate(question, docs, history)
    return answer

History context affects rewriting + answering.

Watch context window: truncate old turns.

Step 10: Production Checklist (15 min)

  • [ ] Hybrid retrieval (not pure vector)

  • [ ] Reranking

  • [ ] Query rewriting for conversations

  • [ ] Citations in output

  • [ ] Refusal for out-of-scope

  • [ ] Latency budget (< 5s ideal)

  • [ ] Cost tracking

  • [ ] Evals before deploy

  • [ ] Logging for investigation

  • [ ] Update pipeline

Each: hours-days of work. Worth it.

What You Just Did

RAG production: naive vs production, retrieval quality, query rewriting, chunking, metadata filters, citations, out-of-scope, updating, conversation memory, checklist.

Common Failure Modes

Naive RAG shipped to prod. Hallucination festival.

No citations. Trust loss.

No refusal mechanism. Always-answer creates lies.

Stale index. Wrong info served.

No evals. Drift unnoticed.

Continue the LLM Application Engineering path

Part of the LLM Application Engineering learning path.

bottom of page