Tutorial 10: Evaluate Prompt Quality

Shawn West

May 30

3 min read

Updated: Jul 13

Without measurement, prompt engineering is guesswork. With it, you can iterate deliberately. This tutorial sets up basic evaluation.

What You'll Build

An evaluation system that scores your prompt's outputs and tracks improvement over versions.

Step 1: Build an Eval Set (1-2 hours)

Real inputs with expected outputs (or properties):

EVAL_CASES = [
    {
        "id": "case_001",
        "input": "Customer says order #12345 hasn't arrived",
        "expected": {
            "category": "shipping",
            "must_contain": ["12345", "delivery"],
            "must_not_contain": ["refund"],  # No refund implied
            "tone": "helpful",
        }
    },
    {
        "id": "case_002",
        "input": "User can't reset their password",
        "expected": {
            "category": "account",
            "must_contain": ["password reset"],
        }
    },
    # ... 20-50 cases total
]

The set covers happy paths, edge cases, things that previously went wrong.

Step 2: Write the Scorer (30 min)

def score_response(response, expected):
    score = {
        "correct_category": expected["category"] in response.lower(),
        "has_required": all(s in response.lower() for s in expected.get("must_contain", [])),
        "lacks_forbidden": not any(s in response.lower() for s in expected.get("must_not_contain", [])),
    }
    
    # Aggregate
    score["pass"] = all(score.values())
    return score

def evaluate(prompt_version, eval_cases):
    results = []
    for case in eval_cases:
        response = call_llm(prompt_version, case["input"])
        score = score_response(response, case["expected"])
        results.append({
            "case_id": case["id"],
            "response": response,
            "score": score,
        })
    
    pass_rate = sum(r["score"]["pass"] for r in results) / len(results)
    return pass_rate, results

Automated scoring catches the obvious.

Step 3: Add Human-in-the-Loop (varies)

Some criteria can't be auto-scored:

Tone
Helpfulness
Correctness of nuanced facts
Writing quality

For these, periodic human review:

def request_human_eval(results, sample_size=10):
    sample = random.sample(results, sample_size)
    
    for result in sample:
        print(f"Input: {result['case_id']}")
        print(f"Response: {result['response']}")
        rating = input("Rate 1-5 (or skip): ")
        result["human_rating"] = rating

Combine auto + human for full coverage.

Step 4: Run on Each Prompt Version (per change)

# Compare versions
v1_pass_rate, _ = evaluate(prompt_v1, eval_cases)
v2_pass_rate, _ = evaluate(prompt_v2, eval_cases)

print(f"v1: {v1_pass_rate:.2%}")
print(f"v2: {v2_pass_rate:.2%}")
print(f"Delta: {(v2_pass_rate - v1_pass_rate) * 100:+.1f}%")

The delta tells you whether the change is an improvement.

Step 5: Track Over Time (15 min)

Log each evaluation:

def log_evaluation(prompt_version, pass_rate, sample_results):
    log = {
        "version": prompt_version,
        "timestamp": datetime.now().isoformat(),
        "pass_rate": pass_rate,
        "sample_failures": [r for r in sample_results if not r["score"]["pass"]][:5],
    }
    
    with open("eval_log.jsonl", "a") as f:
        f.write(json.dumps(log) + "\n")

Build history. See trend.

Step 6: Use LLM-as-Judge (advanced, varies)

For subjective criteria, use an LLM as the evaluator:

def judge_response(response, criteria):
    judge_prompt = f"""
    Evaluate this response against these criteria:
    
    Response: {response}
    
    Criteria:
    {criteria}
    
    Return JSON: {{"pass": true/false, "reasons": [...]}}
    """
    
    result = call_llm(judge_prompt)
    return json.loads(result)

LLM-as-judge is imperfect but scales. Combine with human spot-checks.

Step 7: Group Failures (10 min)

When pass rate drops, find the pattern:

def analyze_failures(results):
    failures = [r for r in results if not r["score"]["pass"]]
    
    # Group by failure type
    from collections import defaultdict
    by_type = defaultdict(list)
    for f in failures:
        for criterion, passed in f["score"].items():
            if not passed:
                by_type[criterion].append(f["case_id"])
    
    return by_type

Patterns guide your fix:

"10 failures all due to tone" → tune the tone instruction
"5 failures across different criteria" → broader regression
"3 failures on specific edge cases" → add few-shot examples

Step 8: A/B Test in Production (varies)

For mature systems:

def get_prompt_for_request(user_id):
    # 90/10 split
    if hash(user_id) % 100 < 10:
        return prompt_v2
    return prompt_v1

# Log which version was used
# Compare outcomes (user satisfaction, downstream metrics)

Production data beats offline eval for some metrics.

Step 9: Build the Eval Dashboard (varies)

Visualize:

Current pass rate per prompt
Trend over time
Failure mode breakdown
Recent changes and their impact

Even a simple dashboard makes the practice durable.

Step 10: Maintain the Eval Set (ongoing)

The eval set evolves:

Add cases when new failure modes appear
Remove cases when behavior is no longer relevant
Update expected outputs when product changes
Periodic review (quarterly)

Stale eval sets stop being useful.

What You Just Did

You set up prompt evaluation. Changes get measured; improvements are real; regressions get caught.

Common Failure Modes

Small eval set. N=3 case results aren't reliable.

Cherry-picked cases. All happy path; production differs.

No human review. Auto-scoring misses subjective issues.

Eval ignored. Numbers exist; behavior doesn't change.

Stale eval set. Doesn't reflect current task.

You're Done

You've completed the prompt engineering path. From setup through evaluation, you have the toolkit to engineer prompts deliberately.

Recommend Building With LLMs Step-by-Step next to apply these prompts in real applications.