top of page

LLM Cost and Latency — LLM Application Engineering, Part 7

S. West
Jul 8
3 min read

Updated: 5 days ago

LLM Application Engineering · Part 7

Corrected, 4 October 2026. We re-checked this post's figures. The price table is now dated (2024 launch prices) and sourced, the per-call cost arithmetic is fixed (the original figure was gpt-4o's cost, not gpt-4o-mini's), and an unsourced quality comparison was removed.

An LLM app's inference bill and its response time are the same problem wearing two hats: both are driven by tokens, and both can quietly balloon as usage grows — until someone forwards a screenshot of the invoice. The good news is there's a well-worn set of levers, and most teams pull them in the wrong order. This walks through cutting cost and latency by impact — picking the cheapest model that works, compressing context, prompt caching, streaming, and batching — with monitoring so you know it actually worked.


LLM apps spend a lot on inference. Each request has cost + latency. Optimize both.


Step 1: Pricing Basics (15 min)


Per million tokens (illustrative 2024 launch prices):


2024 launch prices, shown for the ratio between tiers.
Current prices differ: check each provider's pricing page before you budget.

GPT-4o (Aug 2024):        $2.50 input / $10 output
GPT-4o mini (Jul 2024):   $0.15 input / $0.60 output
Claude 3 Opus (Mar 2024): $15 / $75
Claude 3 Haiku (Mar 2024): $0.25 / $1.25

Output usually 3-4x cost of input.


For chat with long history: input dominates.


For generation tasks: output dominates.


Step 2: Cheapest Model That Works (15 min)


Pattern:


  1. Build with strong model (GPT-4o, Claude Opus)

  2. Eval against test set

  3. Try smaller model

  4. Adjust prompt if needed

  5. If quality holds: ship smaller


Often a smaller model handles the routine share of requests at a fraction of the cost: at 2024 launch prices, gpt-4o-mini's input rate was 6% of gpt-4o's. Measure the quality gap on your own eval set before you switch.


For: every LLM call, ask "smallest model that works?"


Step 3: Prompt Length (10 min)


System prompt: 500 tokens
Retrieved context: 2000 tokens
Conversation history: 1500 tokens
User question: 50 tokens
---
Total input: 4050 tokens

Output: 200 tokens

Each call: ~$0.012 with gpt-4o, or ~$0.0007 with gpt-4o-mini (2024 prices).


At 1M calls / month: about $12,000 with gpt-4o versus about $730 with gpt-4o-mini.


Optimize input length. Cheaper than smaller model often.


Step 4: Context Compression (15 min)


Long convo histories: summarize older turns.


if len(messages) > 10:
    summary = llm.chat("Summarize: " + str(messages[:5]))
    messages = [{'role': 'system', 'content': summary}] + messages[5:]

Keep recent verbatim; older summarized.


For chatbots: extends usable session length without explosion.


Step 5: Caching (15 min)


For repeat queries:


cache_key = hash(prompt + system + model)
if cached := cache.get(cache_key):
    return cached

result = llm.chat(messages)
cache.set(cache_key, result, ttl=3600)
return result

For deterministic temp-0 calls: cache aggressively.


Embedding caching: even more important.


Step 6: Prompt Caching (OpenAI / Anthropic) (15 min)


Provider-side cache for long prompts:


response = openai.chat.completions.create(
  model='gpt-4o',
  messages=[...],
  # prompts hashed; static prefix cached
)

OpenAI launched automatic prompt caching in October 2024 at 50% off cached input tokens; Anthropic bills cache reads at 10% of the input rate. For long system prompts: significant savings.


Anthropic: explicit cache_control markers.


Step 7: Streaming (15 min)


stream = openai.chat.completions.create(stream=True, ...)
for chunk in stream:
    print(chunk.choices[0].delta.content, end='', flush=True)

Perceived latency much lower. User sees response start in <500ms vs waiting 5s.


For chat / generation UX: always stream.


Step 8: Batching (15 min)


OpenAI / Anthropic Batch API:


  • Submit list of requests

  • 24-hour SLA

  • 50% discount


For non-interactive (data processing, batch embedding):


batch = openai.batches.create(
  input_file_id=file_id,
  endpoint='/v1/chat/completions',
)
# wait for completion

For: offline tasks. Big savings.


Step 9: Parallel Calls (15 min)


results = await asyncio.gather(*[
    llm.chat(prompt) for prompt in prompts
])

For independent calls: parallel.


Watch rate limits. Tiered per provider.


For batch processing: 10x speedup possible.


Step 10: Cost Monitoring (15 min)


Track per:


  • Endpoint / feature

  • Customer / tenant

  • User

  • Day / week / month


Alert on:


  • 2x usual rate (anomaly)

  • Specific user > threshold

  • Total approaching budget


Track tokens; bill against business value.


What You Just Did


LLM cost + latency: pricing, cheapest model, prompt length, compression, caching, prompt caching, streaming, batching, parallel, monitoring.


Common Failure Modes


Always biggest model. 10x cost.


No caching. Repeat work.


No streaming. Slow perceived UX.


Batch API unused for offline. Pay full price.


No monitoring. Surprise bill.



Continue the LLM Application Engineering path



Sources



Part of the LLM Application Engineering learning path.

bottom of page