Tutorial 9: Cost-Aware Evaluation

Contributor
4 days ago
3 min read

A 500-case eval, run on every PR, with LLM-as-judge x3 consensus = expensive. Same signal at 10x lower cost is achievable.

Step 1: Measure Current Eval Cost (10 min)

def eval_cost(num_cases, judges_per_case, model, judge_model):
    # Output tokens per case
    output_tokens = 500
    judge_tokens = 200
    
    case_cost = (output_tokens / 1000) * MODEL_COST[model]
    judge_cost = (judge_tokens / 1000) * MODEL_COST[judge_model] * judges_per_case
    
    total_per_case = case_cost + judge_cost
    return total_per_case * num_cases

# Example
# 500 cases * 3 judges * $0.01 = $15/run
# Run on every PR (20/day) = $300/day = $9000/month

Quantify. Often surprises teams.

Step 2: Sample-Based Eval (10 min)

Statistical sampling for CI:

def stratified_sample(eval_set, n=20):
    by_category = group_by_category(eval_set)
    sample = []
    per_category = n // len(by_category)
    for cat, cases in by_category.items():
        sample.extend(random.sample(cases, per_category))
    return sample

20 stratified cases catch the same regressions as 500 random cases, most of the time.

CI: 20 cases. Nightly: full 500.

Step 3: Cheap Filter First (10 min)

Rules cost ~$0; LLM judge costs $0.01:

def grade(output, expected):
    # Cheap rules first
    rule_grade = grade_with_rules(output, expected)
    if not rule_grade["pass"]:
        return rule_grade  # Failed rules; don't bother with LLM judge
    
    # Only spend on LLM judge if rules passed
    return grade_with_llm_judge(output, expected)

Don't pay for LLM judge to confirm what regex caught.

Step 4: Cache LLM Calls (10 min)

import hashlib
from pathlib import Path

CACHE_DIR = Path(".eval-cache")
CACHE_DIR.mkdir(exist_ok=True)

def cached_llm(model, prompt, input_text):
    key = hashlib.sha256(f"{model}|{prompt}|{input_text}".encode()).hexdigest()
    cache_file = CACHE_DIR / f"{key}.json"
    
    if cache_file.exists():
        return json.loads(cache_file.read_text())
    
    result = call_llm(model, prompt, input_text)
    cache_file.write_text(json.dumps(result))
    return result

Hash on (model, prompt, input). Same combo = cached. Re-running an eval on the same prompt is free.

Step 5: Cheaper Judge Model (10 min)

Use a cheaper model for judge if it correlates with the expensive one:

# Test correlation on a sample
expensive_grades = [judge_expensive(c) for c in sample]
cheap_grades = [judge_cheap(c) for c in sample]

correlation = pearson(expensive_grades, cheap_grades)
# If > 0.85, cheap judge works

Haiku as judge for many tasks; Sonnet only for nuanced ones.

Step 6: Reduce Output Tokens (5 min)

LLM judge prompt asking for 5 scores + reasoning = many tokens.

# Verbose
JUDGE = "Evaluate the response and provide detailed reasoning for each metric..."

# Compact
JUDGE = """
Score on 1-5 scale. Output JSON only:
{"helpful": N, "accurate": N, "pass": true/false}
"""

Fewer tokens; same signal.

Step 7: Skip Eval on Trivial Changes (5 min)

on:
  pull_request:
    paths:
      - 'prompts/**'      # Run if prompt changed
      - 'src/llm/**'      # Run if LLM code changed
      # NOT: docs/**, README.md, frontend/**

Saves runs that wouldn't have signal anyway.

Step 8: Quota Budget (5 min)

Set monthly budget for eval:

def check_budget():
    spent = get_monthly_eval_spend()
    budget = 500  # dollars
    
    if spent > budget:
        alert("Eval budget exceeded")
        # Skip non-critical evals; full only weekly

Engineers see budget; helps catch runaway.

Step 9: Parallel Eval Runs (10 min)

from concurrent.futures import ThreadPoolExecutor

def run_eval_parallel(eval_set, prompt, workers=20):
    with ThreadPoolExecutor(max_workers=workers) as executor:
        results = list(executor.map(
            lambda c: eval_case(c, prompt),
            eval_set,
        ))
    return results

100 cases run sequentially: 100 × 2s = 200 seconds. With 20 workers: 10 seconds.

Same cost, faster. Faster = less CI compute time.

Step 10: Trade Off Frequency vs. Depth (5 min)

Match eval depth to risk:

Every commit: rules only, sampled 10 cases (free, fast)
Every PR: rules + cheap LLM judge, 20 cases ($1)
Nightly: full eval + expensive judge, 500 cases ($15)
Pre-release: full eval + 3-judge consensus, all 500 cases ($45)

Cost scales with risk. Most changes don't need pre-release depth.

What You Just Did

Cut eval cost 10x+ while preserving signal. Caching, sampling, layered grading, model selection. Sustainable for high-velocity teams.

Common Failure Modes

Sample too small. Save money, lose signal. 5 cases isn't an eval.

Stratification wrong. Sample misses critical category.

Cache invalidation forgotten. Prompt changed; cache returns old answer.

Cheap judge with no validation. Saves money; produces wrong grades.

Skip CI gates. Save money on the eval that catches regressions.

ShiftQuality