Tutorial 4: Track Prompt Regression Over Time

You changed the prompt. Something got better. Did something also get worse? Track to know.

Step 1: Establish a Baseline (10 min)

Before any change, run your full eval set on the current prompt. Record:

{
  "timestamp": "2026-10-01T10:00:00Z",
  "prompt_version": "v1.0",
  "model": "claude-sonnet-4-6",
  "total_cases": 100,
  "passed": 87,
  "failed": 13,
  "per_case": [
    {"id": "case-001", "passed": true, "score": 4.5},
    ...
  ]
}

This is the baseline. Compare against this.

Step 2: Version Your Prompts (5 min)

Treat prompts like code:

PROMPTS = {
    "v1.0": "You are a helpful assistant. Answer the question concisely.",
    "v1.1": "You are a helpful assistant. Answer concisely and cite sources.",
    "v1.2": "You are an expert assistant. Provide concise, sourced answers.",
}

Or in files: prompts/answer_question_v1.0.txt. Git-tracked.

Step 3: Run Eval on Each Version (15 min)

results_v10 = run_eval(eval_set, prompt=PROMPTS["v1.0"])
results_v11 = run_eval(eval_set, prompt=PROMPTS["v1.1"])
results_v12 = run_eval(eval_set, prompt=PROMPTS["v1.2"])

save_results({
    "v1.0": results_v10,
    "v1.1": results_v11,
    "v1.2": results_v12,
})

Same eval set; different prompts. Apples-to-apples.

Step 4: Compare Pass Rates (5 min)

v1.0: 87/100 = 87%
v1.1: 92/100 = 92% (+5%)
v1.2: 89/100 = 89% (+2%)

v1.1 wins on pass rate. But...

Step 5: Find the Regressions (10 min)

def find_regressions(baseline, new):
    regressed = []
    for case_id in baseline:
        if baseline[case_id]["passed"] and not new[case_id]["passed"]:
            regressed.append(case_id)
    return regressed

Cases that PASSED in v1.0 but FAIL in v1.1.

Even with 92% pass rate, you may have new failures. Find them.

Step 6: Categorize Regressions (10 min)

def categorize(regressions, eval_set):
    by_category = defaultdict(list)
    for case_id in regressions:
        category = eval_set[case_id]["category"]
        by_category[category].append(case_id)
    return by_category

Is the regression in one specific area? Maybe v1.1 fixed some cases but broke a category.

Step 7: Set a Regression Threshold (5 min)

Decide: how many regressions are acceptable?

def is_safe_to_ship(baseline, new):
    regressions = find_regressions(baseline, new)
    new_failures = len(regressions)
    improvements = find_improvements(baseline, new)
    
    if new_failures > 5:
        return False, f"{new_failures} regressions; too many"
    if len(improvements) <= new_failures:
        return False, "Net improvement is zero"
    
    return True, f"+{len(improvements)} improvements, -{new_failures} regressions"

Net wins. Acceptable threshold for regressions.

Step 8: Track Scores, Not Just Pass/Fail (10 min)

Pass/fail is binary; score is continuous:

v10_avg_score = avg(r["score"] for r in v10_results)
v11_avg_score = avg(r["score"] for r in v11_results)

# v1.1 may pass more but score lower (worse quality on the passing ones)

Track both. Continuous metrics catch subtle regressions.

Step 9: Track Across Time (10 min)

Persist results:

import json
from datetime import datetime

def save_eval_run(version, results):
    record = {
        "timestamp": datetime.now().isoformat(),
        "version": version,
        "results": results,
    }
    with open("eval-history.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

Build history. Trends over time.

def plot_trend():
    history = load_history()
    for entry in history:
        print(f"{entry['timestamp']}: {entry['pass_rate']:.1%}")

If pass rate drops over 5 versions, regression.

Step 10: Alert on Regression (5 min)

CI integration:

# In CI pipeline
python eval.py --prompt v1.2 > result.json
python compare.py result.json baseline.json
# Exit 1 if regression beyond threshold

Fail the build; require explicit approval to ship a regression.

What You Just Did

Tracked prompt regression over time. Versioning + automated comparison + thresholds. The eval set keeps you honest as you iterate.

Common Failure Modes

Same prompt, different model. Compare across models too; sometimes the model is the variable.

Eval set drift. You change the eval set; old results no longer comparable.

No versioning. Can't compare; can't rollback.

Local pass, prod failure. Eval set doesn't cover real distribution.

Ignore small regressions. They compound over time.