Tutorial 9: Cost-Monitor an LLM App

Contributor
4 days ago
3 min read

LLM bills can grow fast and quietly. Without monitoring, you discover the cost in the monthly invoice. This tutorial sets up real-time monitoring.

What You'll Build

A monitoring system that tracks LLM costs per user, per feature, per query type — with alerts before things explode.

Step 1: Log Every Call (15 min)

def track_call(call_id, user_id, feature, model, response, latency_ms):
    metrics_db.insert("llm_calls", {
        "call_id": call_id,
        "user_id": user_id,
        "feature": feature,
        "model": model,
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
        "cost_usd": calculate_cost(
            response.usage.input_tokens,
            response.usage.output_tokens,
            model,
        ),
        "latency_ms": latency_ms,
        "timestamp": datetime.now(),
    })

Every call: tokens, cost, who, what feature.

Step 2: Calculate Cost (10 min)

PRICING = {
    "claude-sonnet-4-6": {"input": 3.00, "output": 15.00},  # per 1M tokens
    "claude-opus-4-7": {"input": 15.00, "output": 75.00},
    "gpt-4o": {"input": 2.50, "output": 10.00},
    # ... update with current pricing
}

def calculate_cost(input_tokens, output_tokens, model):
    rates = PRICING.get(model, {"input": 0, "output": 0})
    return (input_tokens * rates["input"] / 1_000_000 +
            output_tokens * rates["output"] / 1_000_000)

Update when prices change.

Step 3: Build the Dashboard (varies)

Metrics to display:

Today:
- Total cost: $XXX
- Calls: XXX
- Average cost per call: $X.XX
- p95 latency: XXX ms

By feature:
- summarize_ticket: $XX (XX% of total)
- generate_alt_text: $XX
- chat: $XX

Top users by cost:
- user_123: $XX (XX calls)
- ...

Cache hit rate: XX%

Use Grafana, Datadog, or build simple.

Step 4: Set Alerts (15 min)

# Hourly check
def check_cost_anomalies():
    last_hour = get_cost_last_hour()
    avg_hourly = get_avg_hourly_cost(days=7)
    
    if last_hour > avg_hourly * 3:
        send_alert(f"LLM cost spike: ${last_hour} (3x normal)")
    
    if today_total() > daily_budget * 0.8:
        send_alert(f"80% of daily LLM budget used")

Alerts before the surprise bill.

Step 5: Set Per-User Limits (15 min)

For B2C or API products:

def check_user_quota(user_id):
    monthly_cost = get_user_monthly_cost(user_id)
    
    if monthly_cost > user.tier_limit:
        raise QuotaExceededError(f"User exceeded ${user.tier_limit}/month")

Apply before the call. Prevents one user from running up enormous bills.

Step 6: Cost Per Outcome (varies)

Cost per call isn't the right metric. Cost per outcome is:

Cost per resolved support ticket
Cost per generated article
Cost per converted user

def feature_unit_economics(feature):
    cost = get_feature_cost(feature)
    successful_uses = get_successful_uses(feature)
    return cost / successful_uses

# Print
print(f"Summary feature: ${feature_unit_economics('summarize'):.2f} per ticket")

Tells you whether features are economically viable.

Step 7: Identify Cost Drivers (varies)

When cost spikes:

Which user? (one user; many users?)
Which feature? (one feature; broadly?)
Which prompt version? (recent change?)
Cache hit rate change?

SELECT feature, model, COUNT(*) as calls, SUM(cost_usd) as total
FROM llm_calls
WHERE timestamp > NOW() - INTERVAL '1 day'
GROUP BY feature, model
ORDER BY total DESC;

Top costs surface.

Step 8: Optimize Once You See Patterns (varies)

Common optimizations:

Switch model: premium for hard tasks, cheaper for simple
Shorten prompts: verbose system prompts cost every call
Cache more: if hit rate is low, more is cacheable
Limit output: if outputs are unnecessarily long
Batch: if many similar calls in close succession

Each optimization targets a specific data finding.

Step 9: Track Cost Trends (ongoing)

Weekly:

Total cost trending up/down?
Cost per call trending up/down?
Cost per outcome trending up/down?

Trends matter more than absolute numbers.

Step 10: Plan for Scale (varies)

If usage doubles, will cost double?

Often yes. But some patterns scale differently:

Per-user caching → costs scale sub-linearly
Bulk operations → linear
Heavy reasoning → can be supra-linear (more complex queries)

Model your scale economics. Adjust pricing/quotas accordingly.

What You Just Did

You have visibility into LLM costs. No more surprise invoices. Optimization decisions are data-driven.

Common Failure Modes

Monthly invoice as monitoring. Already too late.

Aggregate-only. "Total cost is X." Can't drill in.

No alerts. Spike runs for days before noticed.

Cost per call only. Not cost per outcome. Misleads.

No quotas. One user runs up enormous bill.

ShiftQuality