Tutorial 5: Build a Basic RAG System

Contributor
4 days ago
3 min read

RAG (Retrieval-Augmented Generation) grounds an LLM's answers in your own data. The model doesn't make things up; it answers from what you provide. This tutorial builds the core pattern.

What You'll Build

A RAG system: user question → retrieved context → grounded answer.

Step 1: The RAG Loop (5 min)

1. User asks a question
2. Retrieve relevant documents (via embeddings)
3. Build a prompt with the docs as context
4. LLM answers using the context
5. Return answer with source citations

That's it. Everything else is refinement.

Step 2: Prep Your Corpus (15 min)

From Tutorial 4, you have documents with embeddings. For RAG, also split long docs into chunks:

def chunk_document(text, chunk_size=500, overlap=50):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start = end - overlap
    return chunks

# Process each doc
for doc in documents:
    chunks = chunk_document(doc["text"])
    for i, chunk in enumerate(chunks):
        store_chunk({
            "doc_id": doc["id"],
            "chunk_index": i,
            "text": chunk,
            "embedding": embed(chunk),
        })

Chunks are 500 chars with 50-char overlap. Adjust to your domain.

Step 3: Retrieval (10 min)

Get the top-k most similar chunks:

def retrieve_chunks(query, top_k=5):
    query_embedding = embed(query)
    
    with conn.cursor() as cur:
        cur.execute("""
            SELECT doc_id, chunk_index, text, 1 - (embedding <=> %s) as score
            FROM chunks
            ORDER BY embedding <=> %s
            LIMIT %s
        """, (query_embedding, query_embedding, top_k))
        return cur.fetchall()

Step 4: Build the Prompt (15 min)

def build_rag_prompt(question, retrieved_chunks):
    context = "\n\n".join([
        f"[Source {i+1}, doc {c['doc_id']}]\n{c['text']}"
        for i, c in enumerate(retrieved_chunks)
    ])
    
    return f"""
You are a helpful assistant. Answer the question using only the provided sources.

If the sources don't contain the answer, say so.
Cite the source number when stating facts.

Sources:
{context}

Question: {question}

Answer:
"""

The prompt explicitly grounds in the retrieved content.

Step 5: Generate (10 min)

def rag_answer(question):
    chunks = retrieve_chunks(question, top_k=5)
    prompt = build_rag_prompt(question, chunks)
    
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )
    
    return {
        "answer": response.content[0].text,
        "sources": [{"doc_id": c["doc_id"], "chunk": c["chunk_index"]} for c in chunks]
    }

Returns the answer plus sources. Critical for trust.

Step 6: Test It (15 min)

result = rag_answer("How do I reset my password?")
print(result["answer"])
print("Sources:", result["sources"])

Check:

Does the answer make sense?
Does it cite sources?
Are the cited sources actually relevant?
Does it say "I don't know" when appropriate?

Step 7: Handle "Not Found" (10 min)

When retrieval doesn't find relevant content:

def rag_answer(question, min_similarity=0.5):
    chunks = retrieve_chunks(question, top_k=5)
    
    # Check if anything is actually relevant
    if not chunks or chunks[0]["score"] < min_similarity:
        return {
            "answer": "I don't have information about that in my knowledge base.",
            "sources": []
        }
    
    # ... rest

Better to say "I don't know" than hallucinate.

Step 8: Iterate on Chunk Strategy (varies)

Chunking dramatically affects quality:

Too small: chunks lack context
Too large: precision drops; less focused retrieval
Wrong boundaries: cut mid-thought

Experiment. For documentation: 500-1000 chars with section-aware splitting often best. For code: function-level chunks. For chat logs: turn-level.

Step 9: Improve Retrieval Quality (varies)

Common improvements:

Re-ranking: retrieve top-20; rerank with a cross-encoder; keep top-5
Query expansion: generate query variations; retrieve for each
Hybrid search: semantic + keyword (Tutorial 4)
Filtering: by date, doc type, etc., before semantic search

Each improves precision; trade off complexity.

Step 10: Evaluate End-to-End (30 min)

Eval set:

TEST_CASES = [
    {
        "question": "How do I reset my password?",
        "expected_source_id": "doc_pwd_reset",
        "expected_answer_contains": ["click", "email"],
    },
    # ...
]

for case in TEST_CASES:
    result = rag_answer(case["question"])
    
    # Source check
    sources_correct = case["expected_source_id"] in [s["doc_id"] for s in result["sources"]]
    
    # Content check
    content_check = all(
        keyword in result["answer"].lower()
        for keyword in case["expected_answer_contains"]
    )
    
    print(f"Source: {sources_correct}, Content: {content_check}")

Track over time; catch regressions.

What You Just Did

You built a working RAG system. Questions get answered from your data, with citations. The model doesn't have to know everything — it just has to reason over what you give it.

Common Failure Modes

Top-1 retrieval. Misses cases where the answer spans multiple chunks.

No similarity threshold. Returns "answers" even when no relevant content.

No source citation. User can't verify.

Static chunking. One-size-fits-all chunking; some docs degrade badly.

No evaluation. Don't know if it's working.

ShiftQuality