Build an AI Agent, Stage 12: Calibrate the Judge Before You Read Its Score
An LLM judge's score is only as good as its agreement with the people whose judgement it stands in for. Calibrate the judge against those people before reading any score, check it for bias on slices of the data, and promote a new version only when a paired comparison shows it is better.
The example in this series is a composite scenario, built from patterns common to requirements intake in regulated organisations. It is not a single real organisation, and its figures are illustrative: the labels in this stage are composite, not measured. The code is real: every output block below was produced by running it.
Version 2.3 of the Requirements Agent's question skill was meant to make its clarifying questions friendlier and more complete. The team ran the golden set through it and through the current version, 2.2, and scored both with the rubric judge: a model prompted with "Is this a good clarifying question? Answer yes or no." Version 2.3 scored 0.91; version 2.2, 0.86. It was promoted the same afternoon.
Two weeks later the analysts were editing more questions than before, and brokers were taking longer to reply. An analyst pulled twenty of the new drafts. The questions were longer, and more of them ended with an offer: "and would you also like the portal to show the claim history?" That is a leading question, the kind the question skill's own instructions forbid, because it suggests scope the requester never asked for.
The judge had liked them. Nobody had checked whether the judge agreed with the analysts, though the blueprint had asked for exactly that: its evaluation section said question precision would be "judged by a rubric judge calibrated against analysts on 50 drafts". The judge had been written in an afternoon and its score had been read the same day.
This is Stage 12 of Build an AI Agent From Scratch. Stage 11 assembled the test suite and judged its must-never rules. This stage builds the evaluation that decides whether a new version is better: the golden set, the judge, and the bar for promotion.
Why a judge's score feels like a measurement
A model judge turns a subjective question, is this a good clarifying question, into a number that can be computed on 200 cases in minutes and compared between versions. It looks like a measurement, and the research says it can be a good one. A study of LLM judges found that strong judges "can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans" (Zheng et al., 2023).
The same study lists the catches: "position, verbosity, and self-enhancement biases, as well as limited reasoning ability". Verbosity bias is the one that matters here. A judge that prefers longer answers will score a prompt change that makes questions longer as an improvement, whether or not the questions got better. Agreement in a published study is not agreement on your task, with your rubric, against your reviewers. It has to be measured.
Measuring it is also the step most likely to be skipped, because it is the slowest. Labelling fifty drafts takes two analysts an afternoon each. The judge takes minutes to write. And once a score has been read, there is a pull to calibrate the judge until it agrees with the number you already like.
The test: find the judge behind your agent's quality score and ask when it was last compared with a person's labels on the same outputs. If the answer is never, the score measures the judge's taste.
Calibrating against the analysts, slice by slice
Calibration needs people's labels to compare with, and the discovery move is in how they are collected. Two analysts labelled the same fifty question drafts independently, twenty-five short and twenty-five long, as acceptable or not. Then they discussed every draft they disagreed on and wrote down what would you need to see to call this question acceptable? Their answers became the judge's rubric, and the drafts they disagreed on became its worked examples. Their agreement with each other also set the ceiling: no judge can be expected to agree with an analyst more closely than another analyst does.
Agreement is measured with Cohen's kappa, not percentage agreement, because percentage agreement can be high by chance. If most drafts are acceptable, a judge that always answers "acceptable" agrees with the analysts most of the time and tells you nothing. Kappa subtracts the agreement you would expect by chance (Cohen, 1960): 1 is perfect agreement, 0 is no better than chance.
The checker for this stage, eval_check.py, applies three conditions before it will show a judge's scores. Its kappa with each analyst must be at least 0.60 and within 0.10 of the analysts' kappa with each other. And on each slice, short questions and long ones, the judge may not call drafts acceptable more than 10 points more often than the analysts do. If any condition fails, the golden-set scores are withheld.
The original judge:
$ python eval_check.py judge_v1
judge_v1: calibration on 50 drafts labelled by two analysts
analysts agree with each other: kappa 0.84
judge vs analyst A: kappa 0.40 vs analyst B: 0.32 bar 0.60, within 0.10 of the analysts: FAIL
short questions: judge says acceptable 0.68, analysts 0.64 (gap +0.04, limit +0.10): PASS
long questions: judge says acceptable 0.84, analysts 0.36 (gap +0.48, limit +0.10): FAIL
judge NOT calibrated: golden-set scores withheld; fix the judge before reading them
The analysts agree with each other at 0.84. The judge agrees with them at 0.40 and 0.32. And the slice lines show why: on short questions the judge and the analysts are close, but on long questions the judge calls 84% acceptable where the analysts call 36%. That is verbosity bias, measured on this agent's own drafts, and it is exactly the bias that made version 2.3 look better.
The rewritten judge asks the three rules in the question skill's own instructions from Stage 6, each answered yes or no: does the question quote the part of the request it clarifies; does it ask what the requester meant without suggesting what they could also have; can a busy requester answer it in one line? A draft is acceptable only if all three are yes, and six worked examples come from the analysts' disagreements.
$ python eval_check.py judge_v2
judge_v2: calibration on 50 drafts labelled by two analysts
analysts agree with each other: kappa 0.84
judge vs analyst A: kappa 0.80 vs analyst B: 0.80 bar 0.60, within 0.10 of the analysts: PASS
short questions: judge says acceptable 0.72, analysts 0.64 (gap +0.08, limit +0.10): PASS
long questions: judge says acceptable 0.32, analysts 0.36 (gap -0.04, limit +0.10): PASS
golden set (200 requests), clarifying-question precision, bar 0.85
current v2.2: 0.88 candidate v2.3: 0.85
paired difference -0.030, 95% interval [-0.080, +0.020]
verdict: DO NOT PROMOTE: no evidence the candidate is better
(Both from stage12-output.txt. The labels are composite and illustrative; kappa, the slice gaps and the interval are computed by the script.)
Only now does the checker show scores. With a calibrated judge, version 2.3 scores 0.85 against 0.88 for the version it replaced. The uncalibrated judge had scored the same drafts 0.91 against 0.86; a test in the suite confirms it favoured the candidate by five points. The friendlier prompt was rolled back.
The test: split your judge's calibration labels by a property you suspect it is sensitive to, such as length. If its acceptance rate on one slice differs from your people's by more than a few points, the overall agreement is hiding a bias.
A promotion bar that a single run can't clear
The promotion decision has its own rule, and it is stricter than "the new number is higher". Both versions are scored on the same 200 requests, so the comparison is paired: each request contributes the difference between the two versions' results on it, which removes the variation between requests and gives a narrower interval than comparing two separate rates. The checker resamples those 200 differences 2,000 times with a fixed seed and takes the middle 95% as the interval.
A candidate is promoted only if four things hold. The Stage 11 suite says RELEASE, so no must-never rule fails. The judge scoring it is calibrated. The 95% interval on its paired improvement is above zero, so it is shown to be better, not just numerically higher. And its score clears the blueprint's bar of 0.85. Version 2.3 fails the third condition even before its point estimate is considered: the interval runs from −0.08 to +0.02, so there is no evidence it is better, and the best estimate says it is worse.
The rule also protects against the opposite mistake. A candidate that is genuinely better by a small margin may not show it on 200 requests; the answer is a larger golden set or more runs, not a lower bar. Evaluating AI Agents covers sizing the set for the difference you need to detect.
With seven new tests, covering kappa itself, both judges' calibration, the withheld scores, the uncalibrated judge's preference for the candidate, and the promotion rule in both directions, the series suite has 88, and all pass.
The test: take the last version you promoted and compute the paired interval on its improvement over the version before. If the interval includes zero, the promotion was a coin you happened to like.
Evaluating a new version: a method
Have two people label the calibration set independently. Fifty outputs at least, split across slices you suspect matter. Done when you have their kappa with each other.
Build the judge's rubric from their disagreements. Done when the rubric's tests are the questions people asked themselves, and its examples are drafts they disagreed on.
Calibrate before reading any score. Done when the judge's kappa with each person meets the bar and sits close to theirs with each other, and no slice shows a bias beyond the limit.
Score current and candidate on the same golden set. Done when every request has a result for both versions, from the same calibrated judge and the same model version.
Promote on a paired interval above zero. Done when the must-never rules pass, the interval on the primary metric is above zero, the score clears the bar, and no other blueprint metric falls below its own bar.
Tradeoffs and exceptions
Approach | Gains | Costs | Best suited for | Warning signs |
People score every version | The real judgement | Slow; expensive; tiring | Small golden sets; final sign-off | Reviewers scoring hundreds of items in one sitting |
Uncalibrated model judge | Fast, cheap, repeatable | Measures the judge's taste | Nothing you act on | A score nobody has compared with a person |
Calibrated model judge | Fast and trusted, within its calibration | Labelling time; recalibration | Comparing versions on a large set | Calibration done once and never repeated |
Deterministic checks | Exact, free to repeat | Only for what code can check | Quoted spans, schema, hand-overs | Pushing judgement into rules that can't hold it |
Calibration expires. When the judge's model changes, when the rubric changes, or when the agent's outputs change character, as version 2.3's did, the judge's agreement with people has to be measured again. A short monthly re-label of twenty drafts, compared with the judge, catches drift before it reaches a promotion.
Not every metric needs a judge. Of the blueprint's six evaluation metrics, quoted spans and hand-overs are checked deterministically, and relation precision and conflicts are compared with the analysts' labels in the golden set. Only question precision needed a model judge, which is why it was the one place the evaluation could quietly drift. Stage 6 is where each skill got its own bars; this stage makes sure the bar for the question skill is measured by something that agrees with the analysts.
How Stage 12 goes wrong
Calibrating after the score. The judge is tuned until it agrees with a result someone already wants. Calibrate first, and have the tooling withhold scores until it passes, as the checker does.
The judge grades its own model's style. A judge from the same model family as the agent can prefer outputs that sound like itself, the self-enhancement bias. Calibration against people is what exposes it.
One run treated as the comparison. Both versions vary from run to run. Score each version on the same requests, compare them pair by pair, and read the interval, not the two numbers.
The golden set leaks into the prompt. Examples from the golden set end up in the agent's instructions, and the score rises without the agent getting better. Keep the golden set out of every prompt, and refresh it with new closed requests each quarter. Evaluate Agent Reliability covers the mechanics.
What to do next
Ask two people this week to label fifty of your agent's outputs independently, then compute their agreement with each other and with your judge, overall and by length. Don't look at the judge's scores on your golden set until it passes. Stage 13 builds the run record: one trace per run that answers the questions of four different readers, from the analyst to the auditor.
Final takeaway
A judge's score is a measurement only if the judge agrees with the people it stands in for. Calibrate it against two people before reading any score, check it slice by slice for bias, and promote a new version only when a paired comparison shows it is better and its must-never rules hold. Then a higher number means a better agent, not a judge that likes longer questions.
Sources
Lianmin Zheng and others, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks Track. Over 80% agreement between strong LLM judges and human preferences, the same as between humans; position, verbosity and self-enhancement biases.
Jacob Cohen, A Coefficient of Agreement for Nominal Scales, Educational and Psychological Measurement 20(1), 1960. Kappa as chance-corrected agreement.
Series navigation. Previous: Stage 11: A Test Suite Where Must-Never Means Never. Next: Stage 13, a run record that answers four readers from one trace (coming). The full stage list is on the Build an AI Agent From Scratch hub.
Frequently Asked Questions
How do you calibrate an LLM judge?
Have two people label the same set of outputs independently, measure how well they agree with each other, then measure how well the judge agrees with each of them, using a chance-corrected statistic such as Cohen's kappa. Also compare the judge with the people on slices such as long and short outputs, to find biases the overall number hides.
Why use Cohen's kappa instead of percentage agreement?
Because agreement can be high by chance. If most outputs are acceptable, a judge that always says "acceptable" agrees with people most of the time while telling you nothing. Kappa subtracts the agreement you would expect by chance.
When should a new AI agent version be promoted?
When its must-never rules pass, the judge scoring it has been calibrated, and the improvement on the primary metric is clear on a paired comparison, with the 95% interval above zero, rather than a higher number on one run.
Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.
