Token Budgeting — Practical Prompt Engineering, Part 9

Shawn West
Jul 8
11 min read

Updated: Aug 18

Practical Prompt Engineering · Part 9

A token is two things at once, and both of them bite. It's a line item on an invoice — you pay per token in and, at roughly 5x the rate, per token out. And it's a hard wall — the context window is finite, so every token you spend on a bloated system prompt is a token you can't spend on the actual ticket. Most teams meet the bill before they meet the wall: the summarizer we've built across this path runs fine in testing, ships, and three weeks later someone forwards a screenshot of a cloud invoice with a number on it nobody predicted. This tutorial is about predicting that number, then cutting it — by measurement, not by vibes.

The trap is optimizing the wrong thing. People reach for the visible knob (shorten the prompt, cap the output) and leave the biggest lever — prompt caching — untouched, because it's invisible until you look at the usage numbers. So we profile first, cut in order of impact, and re-run evaluate() after every cut so we never trade dollars for a quiet quality regression.

Before you start

workspace.py from Part 1. budget.py imports run, evaluate, cost, log_run, client and MODEL from it, and the entire method depends on evaluate() catching a regression the moment you cut something load-bearing.
Python 3.10+ and an anthropic SDK recent enough to accept cache_control on system blocks and return the cache-creation and cache-read token fields in usage. Without those fields you cannot prove caching fired.
An API key with access to both the model you summarize with and the cheaper model Lever 5 routes simple tickets to.
Your provider's current price sheet open. Every figure here is anchored to the rates quoted in this tutorial, and pricing moves.
Your real system prompt, few-shot block and a representative ticket. You profile actual bytes; the breakdown here is this example's, not yours.
A prompt_log.jsonl with a day of traffic already in it, so before-and-after is a comparison rather than a claim. Block about an hour, and plan to call twice inside the five-minute cache window.

What you'll build

A budget.py that profiles the summarizer per call — system prompt, few-shot, input and output tokens priced separately — then applies five cuts in order of impact. You finish holding a cached-prefix runner that marks the stable system-plus-few-shot block with cache_control, an extended cost_cached() that prices full-rate input, cache writes and cache reads as three distinct classes, and a compressed prompt, capped output and model router that each passed evaluate() before you kept them. The artifact is a prompt whose bill you can attribute line by line, with an eval run proving the summaries didn't get worse.

Profile the summarizer before you touch it

Here is the summarizer from Parts 1–8 as it actually runs in production, per call:

Prompt: ticket_summary   (model: claude-sonnet-5)
- System prompt:      250 tokens
- Few-shot examples:  800 tokens   (4 worked examples)
- Average input:      500 tokens   (the ticket thread)
- Average output:     100 tokens   (the summary)
- Input per call:   1,550 tokens
- Output per call:    100 tokens

Now attach money and volume. At the rates Part 1 put in cost() — $3.00 per million input tokens, $15.00 per million output — one call costs 1,550 × 3/1e6 + 100 × 15/1e6 = $0.00465 + $0.0015 = $0.00615. At 1,000 calls a day that's $6.15/day, about $185/month. Not catastrophic. But look at where the money is: of the 1,550 input tokens, 1,050 are the system prompt plus the few-shot — the exact same bytes, re-sent and re-billed on all 1,000 calls a day. That stable prefix alone is 1,050 × 3/1e6 × 1,000 × 30 ≈ $94/month, and it carries zero new information after the first call. That's the target.

Count tokens with Claude's tokenizer, not a guess

Before cutting anything, measure with the provider's own counter. Do not reach for tiktoken — that's OpenAI's byte-pair encoder, and it undercounts Claude on ordinary English and by more again on code or JSON — enough that a budget built on it is wrong before you start. A budget built on it is wrong before you start. Claude has its own tokenizer, exposed through the count API:

# budget.py — builds directly on the Part 1 workspace
from workspace import run, evaluate, cost, log_run, client, MODEL

def count_tokens(system: str, user: str) -> int:
    """Claude's own tokenizer via the count API — NOT tiktoken."""
    r = client.messages.count_tokens(
        model=MODEL,                       # claude-sonnet-5, one constant, swap here
        system=system,
        messages=[{"role": "user", "content": user}],
    )
    return r.input_tokens

Run it over your real system prompt and a representative ticket and you get the actual numbers above — not an estimate that drifts. Now cut, biggest lever first.

Lever 1: Prompt caching — the one that matters at volume

Caching is the highest-leverage change here and the one people skip, because nothing in the prompt looks different afterward. The idea: the system prompt and few-shot examples are byte-for-byte identical on every call, so ask the provider to keep that prefix warm and bill it at a fraction of the rate. A cache read costs roughly 10% of the normal input rate; a cache write costs about 1.25x (for the 5-minute cache) and happens only on the first call.

You mark the stable prefix with cache_control. Everything volatile — the ticket itself — goes after the cached block, so it never invalidates the cache:

# The stable prefix: system instructions + few-shot, marked cacheable.
SYSTEM_BLOCKS = [
    {"type": "text",
     "text": SYSTEM_PROMPT + "\n\n" + FEW_SHOT_EXAMPLES,
     "cache_control": {"type": "ephemeral"}},
]

def run_cached(user: str, max_tokens: int = 80):
    resp = client.messages.create(
        model=MODEL,
        max_tokens=max_tokens,
        system=SYSTEM_BLOCKS,                 # cached prefix
        messages=[{"role": "user", "content": user}],   # volatile, after the breakpoint
    )
    return resp.content[0].text, resp.usage

Extend Part 1's cost() so it prices the three token classes separately — full-rate input, cache write, cache read:

# Dollars per 1M tokens — CHECK YOUR PROVIDER'S CURRENT PRICING and keep these current.
PRICE_IN_PER_M          = 3.00
PRICE_OUT_PER_M         = 15.00
PRICE_CACHE_WRITE_PER_M = 3.75    # ~1.25x input, paid once when the cache is written
PRICE_CACHE_READ_PER_M  = 0.30    # ~0.10x input — the number that changes the bill

def cost_cached(u) -> float:
    return (u.input_tokens                              * PRICE_IN_PER_M
            + getattr(u, "cache_creation_input_tokens", 0) * PRICE_CACHE_WRITE_PER_M
            + getattr(u, "cache_read_input_tokens", 0)     * PRICE_CACHE_READ_PER_M
            + u.output_tokens                              * PRICE_OUT_PER_M) / 1_000_000

The proof is in usage. On the first call cache_creation_input_tokens is ~1,050 and cache_read_input_tokens is 0; on every call after, it flips — cache_read_input_tokens is ~1,050 and input_tokens drops to just the ~500-token ticket. If cache_read_input_tokens stays 0 across repeated calls, caching silently isn't happening: your prefix is below the minimum cacheable size, or something volatile (a timestamp, an unsorted JSON blob, the ticket text) leaked in ahead of the breakpoint. Diff two rendered prompts byte-for-byte to find it.

The math is the whole argument. That $94/month stable prefix becomes: one write per 5-minute window plus 999-ish reads at 10%. At 1,000 calls/day spread across the day — one roughly every 86 seconds, comfortably inside the 5-minute TTL — the prefix cost falls to about $9–10/month. One change, ~$85/month saved, and it touches nothing the model reads differently, so there is no quality risk to re-test. That last part is why caching outranks every cut below it: the others save less and can regress quality; this one can't.

Lever 2: Compress the system prompt — without quietly regressing quality

The second-biggest lever is also the one most likely to backfire, so it gets the most care. System prompts accrete. Ours opens like this:

You are an extremely helpful and knowledgeable customer service analyst
with extensive experience in handling various types of support tickets.
You will be analyzing a customer support ticket and providing a
comprehensive summary that captures all the key information including but
not limited to the customer's main issue, any actions that have been
taken, and any next steps that may be required.

That's 250 tokens of throat-clearing. The instruction underneath it is three lines:

You are a support analyst. Summarize the ticket:
- The customer's issue
- Actions taken
- Next steps

About 45 tokens — an 80% cut. Same task. The tempting move is to ship it and pocket the savings. Don't ship it on faith. "Comprehensive summary capturing all key information" may have been quietly load-bearing — dropping it can shorten outputs past the point where the no_issue restraint case or the buried-resolution thread still passes. The whole reason Part 1 built evaluate() was so this is a five-second question, not a guess:

SYSTEM_V1 = LONG_SYSTEM_PROMPT      # the 250-token version
SYSTEM_V2 = SHORT_SYSTEM_PROMPT     # the 45-token rewrite

print("=== before ==="); evaluate(SYSTEM_V1)
print("=== after  ==="); evaluate(SYSTEM_V2)

If V2 holds at the same pass count (or better), keep it. If it drops one case, you've learned exactly which sentence was doing work — add that clause back and nothing else. Compressing 205 input tokens/call saves about 205 × 3/1e6 × 30,000 ≈ $18/month uncached. Worth taking — but notice it's a fifth of what caching returned, and it's the one that can silently make the summaries worse. Measure, then cut.

Lever 3: Trim the few-shot — and mind the interaction with caching

Four worked examples cost 800 tokens. Often two do the same job — try dropping to the two that cover your hardest cases (the buried resolution, the no-issue note) and re-run evaluate() exactly as above. If the score holds, you've cut ~400 input tokens/call.

But here's the interaction the naive optimizer misses: caching and compression pull in opposite directions on the few-shot. A cached token already costs ~10% of normal, so once the prefix is cached, the marginal cost of an extra example is tiny — and if you compress the prefix below the minimum cacheable size, you lose caching entirely and every remaining token reverts to full price. For a high-volume prompt you cache, keeping a slightly larger, cache-warm few-shot can be cheaper than a lean, uncached one. So order matters: cache first, then decide whether trimming still pays. Profile, don't assume.

Lever 4: Cap the output with max_tokens

Output tokens are the expensive ones. The summary averages 100 tokens, but "average" hides the tail — the occasional ticket where the model writes five paragraphs and you pay for all of them. max_tokens is a hard ceiling that caps the worst case:

resp = client.messages.create(
    model=MODEL,
    max_tokens=80,          # a 30-word summary fits comfortably; a runaway gets cut
    system=SYSTEM_BLOCKS,
    messages=[{"role": "user", "content": ticket}],
)

This doesn't lower the average much; it kills the tail, which is where surprise output cost hides. Pair it with a "≤ 30 words" instruction in the prompt so the model finishes cleanly instead of getting truncated mid-sentence — a truncated response (stop_reason == "max_tokens") is a bug, not a saving.

Lever 5: Route the easy tickets to a cheaper model

Not every ticket needs the flagship. A one-line password reset and a thank-you note are classification-grade work; a 40-message escalation thread is not. Route by difficulty:

SIMPLE_MODEL = "claude-haiku-4-5"   # cheaper input/output rates — check current pricing

def route(ticket: str) -> str:
    return SIMPLE_MODEL if len(ticket.split()) < 40 else MODEL

If ~40% of tickets are simple, moving them to a model at roughly a third of the input rate trims meaningful cost — but only if quality holds. Run the cheap model through the same evaluate() on your simple-ticket cases first. If it passes, route; if it doesn't, that ticket class stays on claude-sonnet-5. Same discipline as every other lever: the eval decides, not the price sheet.

Re-profile and log the win

Wire the cuts back through Part 1's log so the savings are recorded, not remembered. Call log_run() (now using cost_cached) from inside the loop, then compare a day's prompt_log.jsonl before and after. The stable-prefix line goes from ~$94/month to ~$10; the compressed prompt and output cap shave the rest. What was a ~$185/month prompt lands near $80 — and every dollar of it is attributable to a change you measured, with an eval run proving the summaries didn't get worse.

The five levers, in order

Lever	What it saves	When it backfires
1. Prompt caching	The stable prefix drops to a fraction of its uncached cost, with no quality risk because nothing the model reads changed	The prefix is under the minimum cacheable size, or a volatile value sits ahead of the cache_control breakpoint. Bursty traffic also expires the cache and you pay the write premium again
2. Compress the system prompt	Throat-clearing removed from every single call	The clause you dropped was load-bearing. Outputs shorten past the point where the restraint case or the buried-resolution thread still passes
3. Trim the few-shot	Fewer worked examples carried on every call	Once the prefix is cached a marginal example costs far less. Compress below the minimum cacheable size and you lose caching entirely, reverting every remaining token to full price
4. Cap the output	Kills the expensive tail, where surprise output cost hides	Set too low and stop_reason comes back max_tokens — a truncated summary mid-sentence is a bug, not a saving. Pair it with a word-count instruction
5. Route to a cheaper model	Sends the simple tickets to a cheaper model at a fraction of the input rate	Quality doesn't hold on that ticket class. Run the cheap model through evaluate() on your simple-ticket cases first; if it fails, that class stays put

You're done when

count_tokens() over your own system prompt and a representative ticket prints a real per-call breakdown from the provider's tokenizer — the number you budget against, not an estimate that drifts.
You call run_cached() twice inside the cache window and watch usage flip: the first call reports a cache write with zero reads, the second reports the reverse, with input_tokens dropping to just the ticket. If the read count stays 0 across repeated calls, caching silently isn't happening.
You run evaluate() after every cut and the pass count holds, while a day of prompt_log.jsonl priced with cost_cached comes in lower than the day before it.

Troubleshooting

cache_read_input_tokens is always 0. The cache never hit. Either the prefix is under the provider's minimum cacheable size (caching silently no-ops below it — no error), or a volatile value sits ahead of your cache_control breakpoint. Confirm the ticket text and any timestamps come after the cached block, and diff two rendered prompts to spot a stray dynamic byte.

The bill didn't drop as much as the math said. Your traffic is bursty. The 5-minute cache expires between calls during quiet stretches, so you pay the 1.25x write again on the next burst instead of a 10% read. Check the gap between calls; for spiky traffic, a longer-TTL cache (billed at a higher write multiple) can still win.

Summaries got worse after compressing the prompt. You cut a load-bearing sentence. Re-run evaluate() on V1 vs V2, find the case that broke, and add back only the clause that fixes it — not the whole paragraph.

Truncated summaries ending mid-sentence. max_tokens is too low and stop_reason is max_tokens. Raise the cap a little and add an explicit word limit to the prompt so the model wraps up on its own.

Common mistakes

Optimizing before profiling. Shortening a prompt that's re-sent 1,000×/day saves pennies; caching it saves ~$85/month. You can't tell which is which without the per-call token breakdown — profile first.
Skipping caching because it's invisible. It's the biggest lever at volume and the one with no quality risk. Read usage to confirm it's working; don't assume.
Compressing on faith. Every prompt cut is a hypothesis about quality. Re-run evaluate() after each one, or you're trading a known bill for an unknown regression.
Estimating tokens with tiktoken. It's OpenAI's tokenizer and undercounts Claude — use the provider's count API so your budget matches the invoice.
Uncapped output. One runaway 5,000-token response costs more than a hundred normal ones. max_tokens is a cheap seatbelt.

Continue the Practical Prompt Engineering path

Previous — Part 8: Guardrails — Prompt Injection, Refusals, and Staying On-Task
You've reached the end of the path. Put it all together on a real project, and keep the workspace from Part 1 as your standing harness.