Cache LLM Responses — Building with LLMs, Part 8

Contributor
6 days ago
3 min read

Updated: 4 hours ago

Building with LLMs · Part 8

Cached LLM responses serve instantly and cost nothing. The challenge: knowing what's safe to cache. This tutorial walks through it.

What You'll Build

A caching layer that reduces LLM costs significantly without serving stale results.

Step 1: Identify What's Cacheable (10 min)

Cache when:

Same input → same output expected
Output is not user-specific
Underlying data is stable

Don't cache:

Stateful conversations
User-specific responses
When freshness matters

For RAG over stable docs: highly cacheable. For chat with memory: not cacheable.

Step 2: Pick the Cache Key (15 min)

For a simple prompt:


import hashlib

def cache_key(prompt: str, model: str) -> str:
    h = hashlib.sha256()
    h.update(f"{model}::{prompt}".encode())
    return h.hexdigest()

For a more complex prompt:


def cache_key(system_prompt, user_message, examples, model):
    components = json.dumps({
        "system": system_prompt,
        "user": user_message,
        "examples": examples,
        "model": model,
    }, sort_keys=True)
    return hashlib.sha256(components.encode()).hexdigest()

Anything that affects the output should be in the key.

Step 3: Build the Cache Layer (15 min)


import redis

cache = redis.Redis()

def cached_llm_call(prompt, model="claude-sonnet-4-6", ttl=3600):
    key = cache_key(prompt, model)
    
    cached = cache.get(key)
    if cached:
        return json.loads(cached)
    
    response = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )
    result = response.content[0].text
    
    cache.setex(key, ttl, json.dumps(result))
    return result

Redis with TTL. Hits avoid the LLM call entirely.

Step 4: Track Hit Rate (10 min)


def cached_llm_call(prompt, model="claude-sonnet-4-6"):
    key = cache_key(prompt, model)
    
    cached = cache.get(key)
    if cached:
        metrics.increment("llm_cache.hit")
        return json.loads(cached)
    
    metrics.increment("llm_cache.miss")
    # ... rest

Aim for 30-70% hit rate for cacheable workloads. Below 20% suggests low cache value; above 90% suggests over-caching.

Step 5: Use Provider Caching (varies)

Anthropic and OpenAI offer prompt caching for the prompt itself (not just response):


# Anthropic prompt caching
response = client.messages.create(
    model="claude-sonnet-4-6",
    system=[
        {
            "type": "text",
            "text": large_system_prompt,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": user_query}]
)

System prompt gets cached server-side. Subsequent calls reuse the cached portion. ~10x cost reduction for the cached part.

Use for: large system prompts, fixed few-shot examples, RAG with stable context.

Step 6: Cache RAG Results (15 min)

For RAG:

The retrieval is cacheable (same query → same docs, if docs haven't changed)
The generation is cacheable (same context + question → same answer)


def cached_rag(question, ttl=3600):
    key = cache_key(question, model)
    
    if cached := cache.get(key):
        return json.loads(cached)
    
    chunks = retrieve_chunks(question)
    prompt = build_rag_prompt(question, chunks)
    answer = call_llm(prompt)
    
    result = {"answer": answer, "sources": [c["id"] for c in chunks]}
    cache.setex(key, ttl, json.dumps(result))
    return result

Common questions get fast, cheap answers.

Step 7: Invalidation Strategy (15 min)

When data changes, cache may be stale:


def update_document(doc_id, text):
    # Update doc
    db.update_doc(doc_id, text)
    re_embed(doc_id)
    
    # Invalidate cached RAG answers that might have used this doc
    # (Practical approach: tag cache entries with doc_id; invalidate by tag)
    cache.delete_pattern(f"rag:doc:{doc_id}:*")

Or use shorter TTLs and accept some staleness.

For frequently-changing data, caching may not be worth it.

Step 8: Negative Caching (10 min)

Cache "I don't know" responses too:


def cached_rag(question):
    # ... cache lookup ...
    
    chunks = retrieve_chunks(question)
    if not chunks or chunks[0]["score"] < 0.5:
        # Cache the "not found" too
        result = {"answer": "I don't have info about that.", "sources": []}
        cache.setex(key, ttl=3600, json.dumps(result))
        return result
    
    # ... rest

"I don't know" repeated questions don't pay.

Step 9: Multi-Layer Caching (varies)

For high volume:

L1 (in-process): LRU cache, microseconds
L2 (Redis): milliseconds
L3 (LLM): seconds


from functools import lru_cache

@lru_cache(maxsize=1000)
def lru_cached_call(prompt, model):
    # Check L2
    cached = cache.get(key)
    if cached:
        return cached
    
    # L3 call
    return llm_call(prompt, model)

Each layer adds value at different cost.

Step 10: Monitor and Tune (ongoing)

Track:

Cache hit rate by use case
Total LLM cost (with vs. without cache)
User-perceived latency

Tune TTLs, cache keys, layers based on data.

What You Just Did

You added caching that significantly reduces LLM costs and improves latency. Repeated queries serve from cache; unique queries go to LLM.

Common Failure Modes

Caching personalized responses. User A sees user B's answer.

Stale cache. Data changed; cache didn't invalidate.

No invalidation. TTL is the only mechanism; some data needs explicit invalidation.

Caching errors. Failed calls cached; subsequent users see errors.

Cache key collision. Different inputs → same key. Subtle bug.

Continue the Building with LLMs path

Previous — Part 7: Function Calling and Tool Use
Next — Part 9: Cost-Monitor an LLM App

Part of the Building with LLMs learning path.

ShiftQuality