top of page

LLM Cost and Latency — LLM Application Engineering, Part 7

  • Shawn West
  • Jul 8
  • 3 min read

Updated: Aug 6

LLM Application Engineering · Part 7

An LLM app's inference bill and its response time are the same problem wearing two hats: both are driven by tokens, and both can quietly balloon as usage grows — until someone forwards a screenshot of the invoice. The good news is there's a well-worn set of levers, and most teams pull them in the wrong order. This walks through cutting cost and latency by impact — picking the cheapest model that works, compressing context, prompt caching, streaming, and batching — with monitoring so you know it actually worked.

LLM apps spend a lot on inference. Each request has cost + latency. Optimize both.

Step 1: Pricing Basics (15 min)

Per million tokens:

GPT-4o: $2.50 input / $10 output
GPT-4o-mini: $0.15 input / $0.60 output
Claude Opus: $15 / $75
Claude Haiku: $0.25 / $1.25
Llama 3 70B (Together): $0.90 / $0.90

Output usually 3-4x cost of input.

For chat with long history: input dominates.

For generation tasks: output dominates.

Step 2: Cheapest Model That Works (15 min)

Pattern:

  1. Build with strong model (GPT-4o, Claude Opus)

  2. Eval against test set

  3. Try smaller model

  4. Adjust prompt if needed

  5. If quality holds: ship smaller

Often: gpt-4o-mini does 80% of what gpt-4o does at 10% cost.

For: every LLM call, ask "smallest model that works?"

Step 3: Prompt Length (10 min)

System prompt: 500 tokens
Retrieved context: 2000 tokens
Conversation history: 1500 tokens
User question: 50 tokens
---
Total input: 4050 tokens

Output: 200 tokens

Each call: ~$0.011 with gpt-4o-mini.

At 1M calls / month: $11k.

Optimize input length. Cheaper than smaller model often.

Step 4: Context Compression (15 min)

Long convo histories: summarize older turns.

if len(messages) > 10:
    summary = llm.chat("Summarize: " + str(messages[:5]))
    messages = [{'role': 'system', 'content': summary}] + messages[5:]

Keep recent verbatim; older summarized.

For chatbots: extends usable session length without explosion.

Step 5: Caching (15 min)

For repeat queries:

cache_key = hash(prompt + system + model)
if cached := cache.get(cache_key):
    return cached

result = llm.chat(messages)
cache.set(cache_key, result, ttl=3600)
return result

For deterministic temp-0 calls: cache aggressively.

Embedding caching: even more important.

Step 6: Prompt Caching (OpenAI / Anthropic) (15 min)

Provider-side cache for long prompts:

response = openai.chat.completions.create(
  model='gpt-4o',
  messages=[...],
  # prompts hashed; static prefix cached
)

50% discount on cached prefix. For long system prompts: significant savings.

Anthropic: explicit cache_control markers.

Step 7: Streaming (15 min)

stream = openai.chat.completions.create(stream=True, ...)
for chunk in stream:
    print(chunk.choices[0].delta.content, end='', flush=True)

Perceived latency much lower. User sees response start in <500ms vs waiting 5s.

For chat / generation UX: always stream.

Step 8: Batching (15 min)

OpenAI / Anthropic Batch API:

  • Submit list of requests

  • 24-hour SLA

  • 50% discount

For non-interactive (data processing, batch embedding):

batch = openai.batches.create(
  input_file_id=file_id,
  endpoint='/v1/chat/completions',
)
# wait for completion

For: offline tasks. Big savings.

Step 9: Parallel Calls (15 min)

results = await asyncio.gather(*[
    llm.chat(prompt) for prompt in prompts
])

For independent calls: parallel.

Watch rate limits. Tiered per provider.

For batch processing: 10x speedup possible.

Step 10: Cost Monitoring (15 min)

Track per:

  • Endpoint / feature

  • Customer / tenant

  • User

  • Day / week / month

Alert on:

  • 2x usual rate (anomaly)

  • Specific user > threshold

  • Total approaching budget

Track tokens; bill against business value.

What You Just Did

LLM cost + latency: pricing, cheapest model, prompt length, compression, caching, prompt caching, streaming, batching, parallel, monitoring.

Common Failure Modes

Always biggest model. 10x cost.

No caching. Repeat work.

No streaming. Slow perceived UX.

Batch API unused for offline. Pay full price.

No monitoring. Surprise bill.

Continue the LLM Application Engineering path

Part of the LLM Application Engineering learning path.

bottom of page