Tutorial 2: Automated Grading with Rules

Contributor
4 days ago
3 min read

Rules are deterministic; cheap; don't require another LLM. Use them where they fit.

Step 1: Identify What Rules Can Check (5 min)

Rules work for:

Format: valid JSON, exact field names
Length: within bounds
Content: required substrings, forbidden words
Type: values within allowed set
Structure: correct nesting

Rules don't work for:

Subjective quality (writing style, helpfulness)
Nuanced correctness (factual accuracy beyond keyword match)
Tone

For subjective: see LLM-as-judge (Tutorial 3).

Step 2: Schema Validation (10 min)

For structured output:

from pydantic import BaseModel, ValidationError

class TicketAnalysis(BaseModel):
    category: Literal['billing', 'technical', 'account']
    urgency: Literal['low', 'medium', 'high']
    summary: str

def validate_output(output_text):
    try:
        data = json.loads(output_text)
        TicketAnalysis(**data)
        return {"valid_schema": True}
    except (json.JSONDecodeError, ValidationError) as e:
        return {"valid_schema": False, "error": str(e)}

Hard fail on schema violations. Common LLM failure mode.

Step 3: Length Checks (5 min)

def check_length(text, min_words=10, max_words=200):
    word_count = len(text.split())
    return min_words <= word_count <= max_words

LLMs often produce too long or too short outputs. Catch.

Step 4: Required Content (10 min)

def must_contain(output, required_terms):
    output_lower = output.lower()
    missing = [t for t in required_terms if t.lower() not in output_lower]
    return {"contains_all": len(missing) == 0, "missing": missing}

Output must reference specific facts from the input.

case = {
    "input": "Customer order #12345 didn't arrive",
    "expected": {"must_contain": ["12345"]}
}

If the summary loses the order number, it's wrong.

Step 5: Forbidden Content (5 min)

def must_not_contain(output, forbidden_terms):
    found = [t for t in forbidden_terms if t.lower() in output.lower()]
    return {"no_forbidden": len(found) == 0, "found": found}

For:

PII leakage (no SSN patterns in output)
Profanity
Competitor mentions
Things outside the model's scope

Step 6: Regex Patterns (5 min)

import re

def check_email_format(text):
    return bool(re.match(r'^[^@]+@[^@]+\.[^@]+$', text))

def has_pii(text):
    patterns = {
        "ssn": r'\d{3}-\d{2}-\d{4}',
        "credit_card": r'\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}',
        "phone": r'\d{3}[-.\s]?\d{3}[-.\s]?\d{4}',
    }
    for name, pattern in patterns.items():
        if re.search(pattern, text):
            return {"has_pii": True, "type": name}
    return {"has_pii": False}

Catches accidental PII leakage.

Step 7: Reference Comparisons (10 min)

For tasks with ground truth:

def compare_to_reference(output, reference):
    # Exact match
    if output.strip() == reference.strip():
        return {"exact_match": True}
    
    # Token overlap (BLEU, ROUGE)
    overlap = jaccard_similarity(output, reference)
    return {"overlap": overlap, "exact_match": False}

def jaccard_similarity(a, b):
    set_a = set(a.lower().split())
    set_b = set(b.lower().split())
    intersection = len(set_a & set_b)
    union = len(set_a | set_b)
    return intersection / union if union > 0 else 0

ROUGE for summarization; BLEU for translation. Imperfect; better than nothing.

Step 8: Combine Checks (10 min)

def grade_output(output, expected):
    grades = {}
    
    if "schema" in expected:
        grades.update(validate_schema(output, expected["schema"]))
    
    if "length" in expected:
        grades["length_ok"] = check_length(output, **expected["length"])
    
    if "must_contain" in expected:
        grades.update(must_contain(output, expected["must_contain"]))
    
    if "no_pii" in expected and expected["no_pii"]:
        grades.update(has_pii(output))
        grades["no_pii"] = not grades["has_pii"]
    
    grades["pass"] = all(v for k, v in grades.items() if isinstance(v, bool) and k != "pass")
    
    return grades

Composable; per-case configurable.

Step 9: Format-Aware Checks (5 min)

For specific output types:

def grade_json_output(output, expected_schema):
    try:
        data = json.loads(output)
    except:
        return {"valid": False}
    
    # Check expected fields
    missing = [f for f in expected_schema if f not in data]
    return {
        "valid": True,
        "missing_fields": missing,
        "has_extra": [k for k in data if k not in expected_schema],
    }

Specific to your output format.

Step 10: Limit Rules Where Subjective (5 min)

Don't try to grade "good writing" with rules. You'll fight the tool.

For subjective: use LLM-as-judge (Tutorial 3). For objective: rules win on speed, cost, determinism.

What You Just Did

Automated grading via rules. Cheap. Deterministic. Catches the obvious problems before they reach human review.

Common Failure Modes

Over-strict regex. Real outputs vary; regex too narrow.

Rules for subjective. "Good tone" can't be regex'd.

Forgotten edge cases. Empty outputs, null, unusual encoding.

Slow regex (catastrophic backtracking). Eval grinds to halt on certain inputs.

No clarity on what "pass" means. Different criteria; threshold unclear.

ShiftQuality