Tutorial 6: LLM Evals

Contributor
4 days ago
2 min read

Without evals, prompts are vibes. With evals, improvements are measured.

Step 1: Why Evals (10 min)

You change a prompt.

Better for case A
Worse for case B
Same for case C

Without eval: don't know. Just ship hoping.

Evals: systematic measurement.

Step 2: Test Sets (15 min)

Build a dataset:

[
  {
    "input": "User: I want a refund",
    "expected": { "intent": "refund_request", "urgency": "medium" }
  },
  {
    "input": "User: Help!",
    "expected": { "intent": "general_help", "urgency": "high" }
  }
]

50-200 examples covering typical + edge cases.

Update over time. New failure cases get added.

Step 3: Metric Types (15 min)

Exact match: did output equal expected?
Contains: does output contain expected substring?
Schema match: structured output valid?
Numeric: difference within tolerance?
LLM-judged: another LLM scores quality

Mix per task.

Step 4: LLM-as-Judge (15 min)

For subjective quality:

def judge(input, output, criteria):
    prompt = f"""
    Given input: {input}
    And the response: {output}
    
    Rate the response 1-5 on: {criteria}
    
    Output: <score> <justification>
    """
    return llm.chat(prompt)

For: tone, helpfulness, safety.

Caveats:

Biased by judge model
Expensive (2x calls)
Calibrate against human judgment

Step 5: Pairwise Comparison (15 min)

Compare two responses:

Which response is better for the question?
A: ...
B: ...

Answer A, B, or tie.

Less prone to judge bias than scoring.

For comparing prompt versions.

Step 6: Tools (15 min)

Braintrust: LLM-focused; experiments
LangSmith: LangChain ecosystem
OpenAI Evals: OSS
PromptFoo: OSS; CLI-friendly
Helicone: observability + eval

Pick one. Don't roll entirely your own; pain.

Step 7: Run in CI (15 min)

- run: pytest tests/llm_evals.py
- run: braintrust run my-prompt --threshold 0.85

Block PRs that drop quality.

Eval gates make prompt changes safe.

Step 8: Production Monitoring (15 min)

Beyond pre-deploy:

Sample 5% production traffic
Send to LLM judge
Track quality over time
Alert on drops

Model drift / new query patterns surface.

For prompt regression in production: this catches.

Step 9: User Feedback (10 min)

[Was this helpful?] [👍] [👎]

User feedback signals:

Which queries fail
Drift over time
Specific bug reports

Feed back into eval set.

For: continuous improvement loop.

Step 10: Common Mistakes (15 min)

No eval set. Operate blind.
Eval set too small. Doesn't catch edge cases.
Train on eval set. Overfitting.
Trust LLM judge without calibration. Biased.
Eval only happy path. Edge cases break in prod.

Eval engineering is real engineering.

What You Just Did

LLM evals: why, test sets, metric types, LLM-as-judge, pairwise, tools, CI, production monitoring, user feedback, mistakes.

Common Failure Modes

No evals. Drift; regression.

Eval set stale. Drift mismatched.

LLM judge biased. Wrong rankings.

Skip CI gate. Bad prompt ships.

Production sample too small. Drift hidden.

ShiftQuality