Tutorial 10: Continuous Evaluation in Production

Shawn West
May 31
3 min read

Updated: Jul 13

The model is live. Now what? Continuous eval = catch regression early, build trust, maintain quality.

Step 1: Online Eval Sampling (10 min)

Don't eval every production request. Sample:

import random

def maybe_eval(request, response):
    if random.random() < 0.01:  # 1% sample
        async_evaluate(request, response)

1% of traffic = enough signal at low cost.

Step 2: Async Eval (10 min)

Don't block the user response. Eval in the background:

def handle_request(request):
    response = call_llm(request)
    
    # Return to user immediately
    send_response(response)
    
    # Eval asynchronously
    if should_sample():
        queue_eval_job({"request": request, "response": response})

Background worker processes the queue.

Step 3: Track Quality Metrics (10 min)

def evaluate_production(sample):
    grades = grade_with_rules_and_judge(sample.response)
    
    record_metric("prod_quality.helpfulness", grades["helpfulness"])
    record_metric("prod_quality.accuracy", grades["accuracy"])
    record_metric("prod_quality.passed", 1 if grades["pass"] else 0)

Surface in dashboards. Quality trend over time.

Step 4: User Feedback Loop (10 min)

In-product feedback:

Thumbs up/down on responses
"This was helpful" / "This was unhelpful"
Specific complaints ("not accurate", "too long")

def on_feedback(message_id, rating, reason=None):
    save_feedback({
        "message_id": message_id,
        "rating": rating,
        "reason": reason,
        "variant": get_variant(message_id),
        "model_version": get_model_version(message_id),
    })

Feedback rate = ground truth. Compare to eval grades; calibrate.

Step 5: Anomaly Detection (10 min)

def detect_anomaly(metric_name):
    recent = get_metric_values(metric_name, last_n=24)  # 24 hours hourly
    historical = get_metric_values(metric_name, last_n=24*7)  # 1 week
    
    if mean(recent) < mean(historical) - 2 * std(historical):
        alert(f"{metric_name} dropped 2 sigma below baseline")

Statistical baseline. Alert on deviation.

Step 6: Cohort Analysis (10 min)

Quality may differ by user type:

def quality_by_cohort():
    by_cohort = defaultdict(list)
    for sample in production_samples:
        cohort = get_user_cohort(sample.user_id)  # new vs. power user, segment, etc.
        by_cohort[cohort].append(sample.quality_score)
    
    for cohort, scores in by_cohort.items():
        print(f"{cohort}: {mean(scores):.2f}")

If quality is worse for new users, the prompt may not be onboarding-friendly.

Step 7: Dashboard (15 min)

Quality dashboard:

Pass rate over time (line)
Per-category pass rate (heatmap)
Top failure categories (bar)
User feedback rate (line)
Cost per request (line)
Latency p50/p99 (line)

Single place. Engineers + product see the same numbers.

Step 8: SLO for Quality (10 min)

Like service SLOs:

quality_slo:
  target: 95%  # of responses pass eval
  measurement_window: 7d
  burn_rate_alert: 2x  # burning budget 2x faster than allowed

Production drops below 90% pass rate for a sustained period = page.

Step 9: Human Review Sampling (10 min)

Random sample for human review:

def daily_review_sample(n=20):
    today = get_today_samples()
    sample = random.sample(today, n)
    # Add edge cases too
    sample.extend(get_today_failures()[:10])
    save_for_review(sample)

Reviewers spot-check. Catches things machines miss.

Step 10: Close the Loop on Production Issues (10 min)

When you detect a quality issue:

Investigate (which cases? which users? what changed?)
Reproduce (replay the failures)
Hypothesize (prompt, model, RAG corpus, etc.)
Fix (test in eval first; A/B test in prod)
Add to eval set (prevent regression)
Post-mortem (what could have caught this sooner?)

Every production issue = improvement to eval coverage.

What You Just Did

Continuous evaluation in production. Sampling, anomaly detection, feedback loop, SLO, human review. Eval is no longer a one-time thing; it's how you operate AI products.

Common Failure Modes

No sampling. Eval everything = cost blow-up.

Sample too small. No signal.

Feedback never analyzed. Collected but unread.

No alerts. Quality drops; nobody notices.

No human review. Metrics look fine; users hate it.

You're Done

You've completed Path 20. Eval sets, automated grading, LLM judges, regression tracking, A/B testing, drift monitoring, CI integration, failure replay, cost control, continuous evaluation in production.

You've now finished 20 paths covering Git, SQL, Docker, Kubernetes, observability, CI/CD, system design, performance, security, AI evals — plus the original paths for testing fundamentals, frontend, backend, AI agents, and more.

Recommend revisiting paths matching your current focus, or applying these techniques in a real project.

ShiftQuality