Cache LLM Responses — Building with LLMs, Part 8

S. West

Jul 8

3 min read

Updated: Oct 4

Updated, 4 October 2026. The hit-rate guidance below is now labelled as a rule of thumb.

Building with LLMs · Part 8

Cached LLM responses serve instantly and cost nothing. The challenge: knowing what's safe to cache. This tutorial walks through it.

What You'll Build

A caching layer that reduces LLM costs significantly without serving stale results.

Step 1: Identify What's Cacheable (10 min)

Cache when:

Same input → same output expected
Output is not user-specific
Underlying data is stable

Don't cache:

Stateful conversations
User-specific responses
When freshness matters

For RAG over stable docs: highly cacheable. For chat with memory: not cacheable.

Step 2: Pick the Cache Key (15 min)

For a simple prompt:

import hashlib

def cache_key(prompt: str, model: str) -> str:
    h = hashlib.sha256()
    h.update(f"{model}::{prompt}".encode())
    return h.hexdigest()

For a more complex prompt:

def cache_key(system_prompt, user_message, examples, model):
    components = json.dumps({
        "system": system_prompt,
        "user": user_message,
        "examples": examples,
        "model": model,
    }, sort_keys=True)
    return hashlib.sha256(components.encode()).hexdigest()

Anything that affects the output should be in the key.

Step 3: Build the Cache Layer (15 min)

import redis

cache = redis.Redis()

def cached_llm_call(prompt, model="claude-sonnet-4-6", ttl=3600):
    key = cache_key(prompt, model)
    
    cached = cache.get(key)
    if cached:
        return json.loads(cached)
    
    response = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )
    result = response.content[0].text
    
    cache.setex(key, ttl, json.dumps(result))
    return result

Redis with TTL. Hits avoid the LLM call entirely.

Step 4: Track Hit Rate (10 min)

def cached_llm_call(prompt, model="claude-sonnet-4-6"):
    key = cache_key(prompt, model)
    
    cached = cache.get(key)
    if cached:
        metrics.increment("llm_cache.hit")
        return json.loads(cached)
    
    metrics.increment("llm_cache.miss")
    # ... rest

As a starting heuristic (our rule of thumb, not a benchmark), a hit rate of roughly 30–70% is typical for cacheable workloads. Much below 20% suggests the cache isn't earning its keep; above 90% can mean you're serving stale answers, so check your invalidation.

Step 5: Use Provider Caching (varies)

Anthropic and OpenAI offer prompt caching for the prompt itself (not just response):

# Anthropic prompt caching
response = client.messages.create(
    model="claude-sonnet-4-6",
    system=[
        {
            "type": "text",
            "text": large_system_prompt,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": user_query}]
)

System prompt gets cached server-side. Subsequent calls reuse the cached portion. ~10x cost reduction for the cached part.

Use for: large system prompts, fixed few-shot examples, RAG with stable context.

Step 6: Cache RAG Results (15 min)

For RAG:

The retrieval is cacheable (same query → same docs, if docs haven't changed)
The generation is cacheable (same context + question → same answer)

def cached_rag(question, ttl=3600):
    key = cache_key(question, model)
    
    if cached := cache.get(key):
        return json.loads(cached)
    
    chunks = retrieve_chunks(question)
    prompt = build_rag_prompt(question, chunks)
    answer = call_llm(prompt)
    
    result = {"answer": answer, "sources": [c["id"] for c in chunks]}
    cache.setex(key, ttl, json.dumps(result))
    return result

Common questions get fast, cheap answers.

Step 7: Invalidation Strategy (15 min)

When data changes, cache may be stale:

def update_document(doc_id, text):
    # Update doc
    db.update_doc(doc_id, text)
    re_embed(doc_id)
    
    # Invalidate cached RAG answers that might have used this doc
    # (Practical approach: tag cache entries with doc_id; invalidate by tag)
    cache.delete_pattern(f"rag:doc:{doc_id}:*")

Or use shorter TTLs and accept some staleness.

For frequently-changing data, caching may not be worth it.

Step 8: Negative Caching (10 min)

Cache "I don't know" responses too:

def cached_rag(question):
    # ... cache lookup ...
    
    chunks = retrieve_chunks(question)
    if not chunks or chunks[0]["score"] < 0.5:
        # Cache the "not found" too
        result = {"answer": "I don't have info about that.", "sources": []}
        cache.setex(key, ttl=3600, json.dumps(result))
        return result
    
    # ... rest

"I don't know" repeated questions don't pay.

Step 9: Multi-Layer Caching (varies)

For high volume:

L1 (in-process): LRU cache, microseconds
L2 (Redis): milliseconds
L3 (LLM): seconds

from functools import lru_cache

@lru_cache(maxsize=1000)
def lru_cached_call(prompt, model):
    # Check L2
    cached = cache.get(key)
    if cached:
        return cached
    
    # L3 call
    return llm_call(prompt, model)

Each layer adds value at different cost.