Tutorial 6: Monitor for Prompt Drift

Contributor
4 days ago
2 min read

You didn't change anything. Production quality dropped anyway. The model changed underneath, or the inputs changed. Detect both.

Step 1: Identify Drift Sources (5 min)

Things that drift even when your code doesn't:

Model updates: vendor pushes a new version
Input distribution: users ask different questions
External data: RAG corpus changes
Token costs: vendor pricing change
Latency: vendor infrastructure changes

Each requires different detection.

Step 2: Pin Model Versions (5 min)

# Bad
client.messages.create(model="claude-sonnet")

# Good
client.messages.create(model="claude-sonnet-4-6")

Pin to specific versions. When upgrading, opt in deliberately and run eval.

Step 3: Track Output Distribution (15 min)

Compute statistics on production outputs:

def track_output_stats(outputs):
    return {
        "avg_length": mean(len(o) for o in outputs),
        "length_p50": median(len(o) for o in outputs),
        "length_p99": percentile(99, [len(o) for o in outputs]),
        "format_consistency": pct_match_schema(outputs),
        "common_phrases": top_ngrams(outputs, n=3, k=20),
    }

Per day. If avg length suddenly jumps 30%, something changed.

Step 4: Track Input Distribution (10 min)

def track_input_stats(inputs):
    return {
        "topic_distribution": classify_topics(inputs),
        "avg_length": mean(len(i) for i in inputs),
        "language_distribution": detect_languages(inputs),
    }

If suddenly 30% of inputs are in Spanish but your prompt is English-only, your eval set may be wrong now.

Step 5: Periodic Eval Replay (15 min)

Daily or weekly, run your eval set against production prompts:

def daily_eval_check():
    eval_set = load_eval_set()
    current_prompt = get_production_prompt()
    
    results = run_eval(eval_set, current_prompt)
    
    historical = load_history()
    latest_baseline = historical[-7:]  # last week's runs
    avg_baseline = mean(r["pass_rate"] for r in latest_baseline)
    
    if results["pass_rate"] < avg_baseline - 0.05:
        alert("Pass rate dropped 5% vs. weekly baseline")

Same inputs, same prompt, same model — if results drift, something external moved.

Step 6: Anomaly Detection on Metrics (10 min)

def detect_anomaly(metric_values):
    mean_val = mean(metric_values[:-1])
    std_val = std(metric_values[:-1])
    latest = metric_values[-1]
    
    z_score = (latest - mean_val) / std_val
    return abs(z_score) > 3  # 3 sigma

Alert on:

Pass rate
Avg score
Avg output length
Avg latency
Cost per request

Step 7: Sample Production Outputs for Review (15 min)

Sample randomly:

def daily_sample(n=20):
    today_outputs = load_today_outputs()
    sample = random.sample(today_outputs, n)
    save_for_review(sample)

A human spot-checks. Catches qualitative drift that metrics miss.

Step 8: Track User Feedback Trends (10 min)

def feedback_trend(days=30):
    feedback = load_feedback(days=days)
    daily_rates = group_by_day(feedback)
    
    for day, rate in daily_rates.items():
        print(f"{day}: {rate['positive']/rate['total']:.1%}")

If thumbs-up drops 5% week-over-week, drift.

Step 9: Alerting Rules (10 min)

alerts = {
    "pass_rate_drop": {
        "metric": "daily_eval_pass_rate",
        "condition": "< baseline - 0.05",
        "severity": "high",
    },
    "latency_spike": {
        "metric": "p99_latency",
        "condition": "> baseline * 1.5",
        "severity": "medium",
    },
    "cost_spike": {
        "metric": "daily_cost",
        "condition": "> baseline * 1.3",
        "severity": "medium",
    },
}

Document; route to right team; rehearse responses.

Step 10: Response Playbook (5 min)

When drift detected:

Confirm (re-run eval; check for noise)
Identify cause:
- Model version changed?
- Input distribution shift?
- Vendor incident?
Mitigate:
- Roll back to known-good prompt
- Pin to older model
- Patch prompt for new input distribution
Post-mortem
Update detection if you missed signal

What You Just Did

Production drift detection. Eval replay + distribution tracking + anomaly alerts. The model doesn't ask permission to change; you catch it.

Common Failure Modes

No baseline. Can't detect "different" without a reference.

Floating model. No version pin; vendor updates ship to you silently.

Eval set rot. Eval set no longer represents prod traffic.

Alert fatigue. Too many alerts; signal lost.

Reactive only. Catch drift at +20% regression; should've caught at +5%.

ShiftQuality