Tutorial 9: Token Budgeting

Contributor
4 days ago
3 min read

Tokens are both a cost (you pay per token) and a constraint (context windows are finite). This tutorial walks through managing token budgets.

What You'll Build

A token-aware prompt system that fits within budgets without sacrificing what matters.

Step 1: Understand Token Pricing (5 min)

Approximate pricing (varies; check current):

Premium models: $3-15 per million input tokens, $15-75 per million output
Mid-tier: $0.50-3 per million input, $1.50-15 per million output
Cheaper: $0.10-0.50 per million

Output is typically 5x more expensive than input.

Step 2: Count Tokens (10 min)

For each prompt:

import tiktoken

def count_tokens(text, model="gpt-4"):
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))

# Or anthropic's tokenizer for Claude
# from anthropic import Anthropic
# client.count_tokens(text)

Know how many tokens your prompts consume. Per-call cost is predictable.

Step 3: Profile Your Prompts (15 min)

For each prompt in production:

Prompt: ticket_summary
- System prompt: 250 tokens
- Few-shot examples: 800 tokens
- Average input: 500 tokens
- Average output: 100 tokens
- Total per call: ~1650 tokens
- Calls per day: 1000
- Daily cost: ~$5
- Monthly: ~$150

The numbers reveal where to optimize.

Step 4: Compress the System Prompt (15 min)

System prompts often have bloat:

Before:

You are an extremely helpful and knowledgeable customer service 
analyst with extensive experience in handling various types of 
support tickets. You will be analyzing a customer support ticket 
and providing a comprehensive summary that captures all the key 
information including but not limited to the customer's main issue, 
any actions that have been taken, and any next steps that may be 
required.

After:

You are a support analyst. Summarize tickets:
- Customer's issue
- Actions taken
- Next steps

Same task, 80% fewer tokens. Often equivalent output quality.

Step 5: Trim Few-Shot Examples (15 min)

Examples are token-heavy. Each example: 100-500 tokens.

Strategies:

Use minimal examples (sometimes 1-2 is enough)
Shorten the examples
Use dynamic example selection (only include relevant ones per input)
Use one example with explanation instead of three without

Measure if removing an example degrades quality. Often it doesn't.

Step 6: Compress Input Context (varies)

For long context (documents, etc.):

Chunk and retrieve: only include the relevant portion (RAG pattern)
Summarize first: if you have a long doc, summarize it before using in the prompt
Strip noise: remove navigation, footers, repeated boilerplate

A 10k-token document might be answerable from a 1k-token excerpt.

Step 7: Control Output Length (10 min)

Output tokens are expensive. Constrain:

Maximum 50 words.

Or in API:

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=200,  # Hard limit
    ...
)

Lower max_tokens = lower cost (caps the bill).

Step 8: Use Cheaper Models Where Possible (10 min)

For simple tasks:

Classification: cheap models often work
Simple extraction: cheap models work
Reformatting: cheap models work

For complex reasoning, you need the premium model.

Try the cheaper model first. Measure quality. If acceptable, save 80% on cost.

Step 9: Cache What's Stable (15 min)

If your system prompt is the same across many calls:

Anthropic's prompt caching (cache the system prompt + examples)
OpenAI's prompt caching
Your own cache layer

# Anthropic prompt caching example
response = client.messages.create(
    model="claude-sonnet-4-6",
    system=[
        {
            "type": "text",
            "text": large_system_prompt,
            "cache_control": {"type": "ephemeral"}  # Cache this
        }
    ],
    messages=[...]
)

Cached prompts cost ~10% of normal price. Huge savings for high-volume.

Step 10: Track in Production (ongoing)

Per call, log:

Tokens in
Tokens out
Cost
Prompt version

Aggregate daily/weekly:

Total cost
Cost per outcome (per useful action)
Anomalies (sudden spike)

The data drives optimization decisions.

What You Just Did

You built token discipline. Your prompts are concise; you know what they cost; you can optimize specifically rather than uniformly.

Common Failure Modes

Verbose system prompts. Pay for the same tokens millions of times.

No measurement. Surprised by the bill.

Premium model for simple tasks. Overpaying.

No caching. Re-sending the same context every call.

Output uncapped. Model writes 5000 tokens; you pay for all of them.

ShiftQuality