Tutorial 7: Build an Eval CI Pipeline

Manual eval = forgotten eval. CI runs it on every PR. Quality gates apply.

Step 1: CLI for Eval (10 min)

Make eval runnable from command line:

# eval.py
import argparse, json

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--prompt-file", required=True)
    parser.add_argument("--eval-set", required=True)
    parser.add_argument("--output", default="results.json")
    parser.add_argument("--threshold", type=float, default=0.85)
    args = parser.parse_args()
    
    prompt = open(args.prompt_file).read()
    eval_set = json.load(open(args.eval_set))
    
    results = run_eval(prompt, eval_set)
    
    json.dump(results, open(args.output, "w"))
    
    if results["pass_rate"] < args.threshold:
        print(f"FAIL: {results['pass_rate']:.1%} < {args.threshold:.1%}")
        exit(1)
    
    print(f"PASS: {results['pass_rate']:.1%}")

if __name__ == "__main__":
    main()

CLI exits non-zero on failure. CI uses exit code.

Step 2: Detect Prompt Changes (5 min)

Only run eval when prompts change:

# .github/workflows/eval.yml
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'eval/**'
      - 'src/llm/**'

Saves time and API cost.

Step 3: GitHub Actions Workflow (15 min)

name: LLM Eval

on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'eval/**'

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install deps
        run: pip install -r requirements.txt
      
      - name: Run eval (baseline)
        run: |
          git checkout origin/main -- prompts/answer.txt
          python eval.py --prompt-file prompts/answer.txt --eval-set eval/cases.json --output baseline.json
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      
      - name: Run eval (PR)
        run: |
          git checkout HEAD -- prompts/answer.txt
          python eval.py --prompt-file prompts/answer.txt --eval-set eval/cases.json --output pr.json
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      
      - name: Compare
        run: python compare.py baseline.json pr.json
      
      - name: Upload artifacts
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: |
            baseline.json
            pr.json

Baseline vs. PR comparison. Block on regression.

Step 4: Sample-Based Eval (5 min)

Full eval may be slow (100s of cases × multiple seconds each). Sample for speed:

# In CI
python eval.py --prompt-file prompts/answer.txt \
    --eval-set eval/cases.json \
    --sample 20 \
    --output pr.json

20 cases instead of 100. Fast CI, fuller eval nightly.

Step 5: Comment on PR (10 min)

- name: Comment results on PR
  uses: actions/github-script@v7
  with:
    script: |
      const fs = require('fs');
      const results = JSON.parse(fs.readFileSync('pr.json'));
      const baseline = JSON.parse(fs.readFileSync('baseline.json'));
      
      const body = `## Eval Results
      
      | Metric | Baseline | PR | Delta |
      |--------|----------|----|----|
      | Pass rate | ${baseline.pass_rate} | ${results.pass_rate} | ${results.pass_rate - baseline.pass_rate} |
      | Avg score | ${baseline.avg_score} | ${results.avg_score} | ${results.avg_score - baseline.avg_score} |
      | Regressions | - | ${results.regressions.length} | - |
      `;
      
      github.rest.issues.createComment({
        issue_number: context.issue.number,
        owner: context.repo.owner,
        repo: context.repo.repo,
        body,
      });

PR shows the eval results inline. Decision-makers see them.

Step 6: Cache LLM Responses (10 min)

Eval = same inputs, same prompts, same model = cacheable.

import hashlib

def cached_call(prompt, input_text):
    key = hashlib.md5(f"{prompt}{input_text}".encode()).hexdigest()
    cache_file = f".eval-cache/{key}.json"
    
    if os.path.exists(cache_file):
        return json.load(open(cache_file))
    
    result = call_llm(prompt, input_text)
    json.dump(result, open(cache_file, "w"))
    return result

Cache + CI artifact for reuse. Saves time and money on re-runs.

Step 7: Nightly Full Eval (5 min)

on:
  schedule:
    - cron: '0 2 * * *'

jobs:
  full-eval:
    # Full 500-case eval on production prompt

Catches drift even without code changes.

Step 8: Eval Set in Repo (10 min)

eval/
├── cases.json
├── categories/
│   ├── happy-path.json
│   ├── edge-cases.json
│   └── failures.json
└── README.md

Versioned with the code. Each case has expected behavior + tags. Easy to maintain.

Step 9: Cost Tracking (5 min)

def track_eval_cost(num_calls, model):
    cost_per_call = COSTS[model]
    total = num_calls * cost_per_call
    print(f"Eval cost: ${total:.2f}")
    
    # Add to GitHub summary
    with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
        f.write(f"\n## Eval Cost\n${total:.2f}\n")

Engineers see cost; helps catch runaway runs.

Step 10: Block Merge on Regression (5 min)

In GitHub: Settings → Branches → Require status check eval to pass.

PR can't merge if eval CI fails. Hard gate.

What You Just Did

Eval CI pipeline. Runs on prompt changes. Compares baseline vs. PR. Comments on PR. Blocks bad merges.

Common Failure Modes

No threshold; every PR shows results but nothing blocks. Theater.

Cost runaway. Full eval on every commit = hundreds of dollars.

Flaky eval. Noisy LLM outputs; runs fail randomly. Use median of N runs.

Eval set drift. Eval doesn't reflect prod traffic anymore.

No cache. Re-running eval on the same prompt costs 10x what it should.