top of page

Track Prompt Regression Over Time — AI Evaluation & Monitoring, Part 4

S. West
Jul 8
3 min read

Updated: 4 days ago

AI Evaluation & Monitoring · Part 4

Corrected, 4 October 2026. We re-checked this post's figures. Pass-rate changes are now shown as percentage points against a named baseline.

You changed the prompt. Something got better. Did something also get worse? Track to know.


Step 1: Establish a Baseline (10 min)


Before any change, run your full eval set on the current prompt. Record:


{
  "timestamp": "2026-10-01T10:00:00Z",
  "prompt_version": "v1.0",
  "model": "claude-sonnet-4-6",
  "total_cases": 100,
  "passed": 87,
  "failed": 13,
  "per_case": [
    {"id": "case-001", "passed": true, "score": 4.5},
    ...
  ]
}

This is the baseline. Compare against this.


Step 2: Version Your Prompts (5 min)


Treat prompts like code:


PROMPTS = {
    "v1.0": "You are a helpful assistant. Answer the question concisely.",
    "v1.1": "You are a helpful assistant. Answer concisely and cite sources.",
    "v1.2": "You are an expert assistant. Provide concise, sourced answers.",
}

Or in files: prompts/answer_question_v1.0.txt. Git-tracked.


Step 3: Run Eval on Each Version (15 min)


results_v10 = run_eval(eval_set, prompt=PROMPTS["v1.0"])
results_v11 = run_eval(eval_set, prompt=PROMPTS["v1.1"])
results_v12 = run_eval(eval_set, prompt=PROMPTS["v1.2"])

save_results({
    "v1.0": results_v10,
    "v1.1": results_v11,
    "v1.2": results_v12,
})

Same eval set; different prompts. Apples-to-apples.


Step 4: Compare Pass Rates (5 min)


v1.0: 87/100 = 87%
v1.1: 92/100 = 92% (+5 points vs v1.0)
v1.2: 89/100 = 89% (+2 points vs v1.0, -3 vs v1.1)

v1.1 wins on pass rate. But...


Step 5: Find the Regressions (10 min)


def find_regressions(baseline, new):
    regressed = []
    for case_id in baseline:
        if baseline[case_id]["passed"] and not new[case_id]["passed"]:
            regressed.append(case_id)
    return regressed

Cases that PASSED in v1.0 but FAIL in v1.1.


Even with 92% pass rate, you may have new failures. Find them.


Step 6: Categorize Regressions (10 min)


def categorize(regressions, eval_set):
    by_category = defaultdict(list)
    for case_id in regressions:
        category = eval_set[case_id]["category"]
        by_category[category].append(case_id)
    return by_category

Is the regression in one specific area? Maybe v1.1 fixed some cases but broke a category.


Step 7: Set a Regression Threshold (5 min)


Decide: how many regressions are acceptable?


def is_safe_to_ship(baseline, new):
    regressions = find_regressions(baseline, new)
    new_failures = len(regressions)
    improvements = find_improvements(baseline, new)
    
    if new_failures > 5:
        return False, f"{new_failures} regressions; too many"
    if len(improvements) <= new_failures:
        return False, "Net improvement is zero"
    
    return True, f"+{len(improvements)} improvements, -{new_failures} regressions"

Net wins. Acceptable threshold for regressions.


Step 8: Track Scores, Not Just Pass/Fail (10 min)


Pass/fail is binary; score is continuous:


v10_avg_score = avg(r["score"] for r in v10_results)
v11_avg_score = avg(r["score"] for r in v11_results)

# v1.1 may pass more but score lower (worse quality on the passing ones)

Track both. Continuous metrics catch subtle regressions.


Step 9: Track Across Time (10 min)


Persist results:


import json
from datetime import datetime

def save_eval_run(version, results):
    record = {
        "timestamp": datetime.now().isoformat(),
        "version": version,
        "results": results,
    }
    with open("eval-history.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

Build history. Trends over time.


def plot_trend():
    history = load_history()
    for entry in history:
        print(f"{entry['timestamp']}: {entry['pass_rate']:.1%}")

If pass rate drops over 5 versions, regression.


Step 10: Alert on Regression (5 min)


CI integration:


# In CI pipeline
python eval.py --prompt v1.2 > result.json
python compare.py result.json baseline.json
# Exit 1 if regression beyond threshold

Fail the build; require explicit approval to ship a regression.


What You Just Did


Tracked prompt regression over time. Versioning + automated comparison + thresholds. The eval set keeps you honest as you iterate.

For the decision the gate makes on each pull request, including how many runs each case needs and which failures should block, see Prompt Regression Testing in CI.


Common Failure Modes


Same prompt, different model. Compare across models too; sometimes the model is the variable.


Eval set drift. You change the eval set; old results no longer comparable.


No versioning. Can't compare; can't rollback.


Local pass, prod failure. Eval set doesn't cover real distribution.


Ignore small regressions. They compound over time.


To see where your own AI feature is thinnest, the AI Readiness Assessment asks twenty-four questions across evaluation, guardrails, observability and governance.


Continue the AI Evaluation & Monitoring path



Part of the AI Evaluation & Monitoring learning path.

bottom of page