Tutorial 1: Build Your First Eval Set

Shawn West
May 30
3 min read

Updated: 4 days ago

Without an eval set, "improvements" are vibes. With one, you measure deliberately.

Step 1: Define the Task (5 min)

Specific. Not "make the LLM better." Try:

"Summarize support tickets in 2-3 sentences"
"Classify emails as spam/promotional/personal/work"
"Extract structured fields from invoices"

Each has a clear right and wrong.

Step 2: Pick the Cases (15 min)

Aim for 20-50. Cover:

Happy path (common cases): ~50% of set
Edge cases (unusual inputs): ~30%
Failure modes (where it's gone wrong before): ~20%

Source cases from real usage, not made-up examples. Real data has the messy distribution that matters.

Step 3: Define "Correct" (15 min)

For each case, what makes the output correct?

Multi-choice:

{
  "input": "Email subject: 'You won a million dollars'",
  "expected_class": "spam",
}

Exact match expected.

Open-ended:

{
  "input": "Support ticket text...",
  "expected_summary_must_contain": ["refund", "order #12345"],
  "expected_length": (50, 200),  # word count range
  "expected_tone": "professional",
}

Properties the output should have.

Step 4: Capture Edge Cases (10 min)

Edge cases:

Empty input
Very long input
Multiple languages
Edge cases of categorization (genuinely ambiguous)
Inputs with sensitive content

These are where models often go wrong. Include them deliberately.

Step 5: Format as Data (5 min)

# eval_cases.py
EVAL_CASES = [
    {
        "id": "case_001",
        "input": "Customer wrote: 'My order hasn't arrived...'",
        "expected": {
            "category": "shipping",
            "must_contain": ["order"],
            "must_not_contain": ["refund"],
        },
        "notes": "Common case; should NOT assume refund needed",
    },
    # ... more cases
]

Each case is data. Easy to iterate.

Step 6: Write the Scorer (15 min)

Per-case scoring function:

def score(output, expected):
    score = {}
    
    if "category" in expected:
        score["category"] = expected["category"] in output.lower()
    
    if "must_contain" in expected:
        score["contains"] = all(s.lower() in output.lower() 
                                 for s in expected["must_contain"])
    
    if "must_not_contain" in expected:
        score["no_forbidden"] = not any(s.lower() in output.lower() 
                                         for s in expected["must_not_contain"])
    
    score["pass"] = all(v for k, v in score.items() if k != "pass")
    return score

Returns per-criterion + overall pass.

Step 7: Run Eval (10 min)

def run_eval(cases, get_output_fn):
    results = []
    for case in cases:
        output = get_output_fn(case["input"])
        score = score_output(output, case["expected"])
        results.append({
            "case_id": case["id"],
            "output": output,
            "score": score,
        })
    
    pass_rate = sum(r["score"]["pass"] for r in results) / len(results)
    return pass_rate, results

pass_rate, results = run_eval(EVAL_CASES, my_summarizer)
print(f"Pass rate: {pass_rate:.2%}")

Headline number; per-case detail.

Step 8: Investigate Failures (15 min)

failures = [r for r in results if not r["score"]["pass"]]
for f in failures[:5]:
    print(f"CASE {f['case_id']}")
    print(f"OUTPUT: {f['output']}")
    print(f"SCORE: {f['score']}")
    print()

Read the failures. Patterns?

All fail on tone → tune tone instruction
All fail one criterion → that criterion is mis-defined or model can't handle it
Random failures → tighten prompt or change model

Step 9: Iterate (varies)

Adjust prompt
Re-run eval
Compare pass rate

v1_pass, _ = run_eval(EVAL_CASES, summarizer_v1)
v2_pass, _ = run_eval(EVAL_CASES, summarizer_v2)
print(f"v1: {v1_pass:.2%} → v2: {v2_pass:.2%}")

Did v2 actually improve? Numbers say.

Step 10: Maintain the Eval Set (ongoing)

The eval set evolves:

Add cases when new failures appear
Remove cases when behavior is no longer relevant
Update expected when product intent changes
Periodic review (quarterly)

Stale eval sets stop being useful.

What You Just Did

You have an eval set. The foundation for everything else in this path.

Common Failure Modes

Too few cases. N=5 isn't reliable signal.

Made-up cases. Don't match real distribution.

No edge cases. Pass rate looks good; production fails.

Bias in selection. All easy cases; no signal.

Vague expected. Can't score automatically.

Next Tutorial

Automate scoring: Tutorial 2: Automated Grading with Rules.

ShiftQuality