LLM Evaluation Frameworks: Measuring What You Cannot See

Evaluation is treated as a measurement problem: pick your dimensions, pick a scorer, get a number. The harder problem is the one almost nobody runs — establishing that the number is measuring anything at all.

Penhale Mutual's claims-triage assistant reads a first-notice-of-loss report and drafts a summary plus a recommended handling path. It had an eval suite: 240 cases, six dimensions, an LLM-as-judge scoring each output 1–5. A prompt revision moved the average from 3.9 to 4.2, which cleared the release threshold, and the change shipped on a Tuesday.

Adjuster escalations rose the following week. Not dramatically — a handful a day, all of them variations on the summary left out the thing that mattered.

The prompt change had made the assistant more thorough. Longer summaries, more hedging, more covered ground. The judge liked longer answers. It had always liked longer answers, on every run, since the suite was written — and because nobody had ever compared the judge's scores against a human's, the preference had been sitting inside every number the team had ever made a decision with.

The unit is one scored output, and the score is a claim you can be wrong about

This is the layer that sits underneath the rest of testing an AI feature. Where a bug lives is a coverage question; which failures deserve the budget is a prioritisation question; what stops a release is an enforcement question. All three consume eval scores. None of them check whether the scores are real.

So the unit here is narrow on purpose: one output, one dimension, one score. That score is an empirical claim — a competent human reviewing this output for this property would say roughly this. Like any claim, it can be false, and it can be false in a consistent direction, which is the dangerous kind. A noisy judge makes your metric jumpy and you notice. A biased judge makes your metric smooth, confident, and wrong, and you promote on it for eight months.

The test you can run: take the last eval score that changed a decision on your team. Ask what evidence exists that a human would have scored it the same way. If the answer is "the judge prompt says to," you have a number, not a measurement.

A judge you have not calibrated is a confident random number

Calibration is one afternoon of work and it is the difference between a metric and a ritual. Penhale ran it after the escalations:

1. Sample 60 outputs from real production traffic, stratified so the hard cases aren't crowded out by the easy ones.

2. Have two people score each one independently, on the same rubric the judge uses.

3. Where the two humans disagree, resolve it and note why — those notes are the rubric's real bugs.

4. Compare the judge to the resolved human score. Report agreement per dimension, and look at the direction of the misses.

Dimension	Judge/human agreement	Where it went wrong
Faithfulness	88%	Reliable. This is what a judge is genuinely good at
Correctness	84%	Missed domain errors — a wrong policy code reads as fluent
Format	96%	Nearly free. Should have been a regex, not a model call
Completeness	61%	Scored length. Every miss was a long answer scored high
Relevance	73%	Rewarded restating the question before answering it
Harmlessness	91%	Fine on obvious violations, untested on subtle ones

Two things fall out of a table like this, and neither is available from the eval average.

The first is that the aggregate was never the problem — one dimension was. Completeness at 61% is barely better than a coin flip on a five-point scale, and it was being averaged in with a 96% that a regular expression could have produced for free. The average looked healthy because five honest dimensions were carrying one broken one.

The second is that the misses had a shape. Every completeness error ran the same direction: longer output, higher score. That is not noise you can average away by running more cases — running more cases makes a biased estimate more precise, not more correct. It made the metric worse, because it made the team more confident.

The test: score 50 outputs by hand and compute agreement per dimension. Any dimension under about 75% is not measuring what its name says. Then look at the disagreements as a set — if they lean one way, you have a bias, and more eval volume will not help.

Dimensions have to be able to disagree, or you have one dimension

The usual advice is to score several dimensions rather than collapsing to a single quality number, and it's right, but it's incomplete in a way that matters. Splitting a score into six names does nothing if all six move together.

The point of separate dimensions is to catch trades — the change that buys correctness with faithfulness, or thoroughness with relevance. A dimension earns its place only if you can describe an output that scores high on it and low on another. If you can't, it's a synonym, and you're paying a model call to compute the same number twice.

Penhale's set had this problem in both directions. Completeness and relevance moved together on almost every case, because both were being read as "how much did it cover" — while the trade the business actually cared about, thoroughness against usable brevity, had no dimension pointing at it at all. The fix wasn't a better judge prompt. It was replacing completeness with a checklist: for this claim type, are these four facts present, yes or no. Agreement went to 94%, because it stopped asking a language model to have a taste and started asking it to look something up.

The test: for each pair of dimensions, write one output that would score 5 and 2. If you can't, delete one of them — and check whether the tradeoff your business actually argues about has a dimension at all.

The eval set is a sample, and somebody chose it

An eval set is a claim about the input distribution, and eval sets are almost never built by sampling it. They're built by people writing down inputs they thought of, which produces a set weighted toward the inputs that are easy to imagine.

Penhale's 240 cases had been assembled over a year, mostly from bugs. That sounds disciplined and it produced a specific distortion: the set was dense in past failures and thin in the ordinary traffic those failures had been exceptions to. Water damage — 4% of real claims — was 22% of the eval set, because water damage had caused two incidents. Meanwhile the single most common claim type was represented by six cases.

Both halves of the fix are cheap. Stratify the set against real traffic so the common paths are actually covered, and keep the failure-derived cases as a separate named slice that you report on separately rather than blending into an average that neither describes.

The test: put your eval set's inputs next to a week of production traffic, bucketed the same way. Any bucket over-represented by more than about 3× is bending your average; any real bucket at zero is invisible to every score you produce.

When a score is the wrong instrument

Scoring is the default move and it is genuinely wrong for some things:

• Anything with a zero-tolerance failure. Cross-tenant leakage doesn't want a 4.6 average; it wants a pass/fail assertion that blocks on one occurrence. Averaging is precisely how a rare catastrophic failure gets smoothed into an acceptable-looking number.

• Anything a cheaper check settles. Format at 96% agreement was a schema validation wearing a model's costume. If a regex, a parser or an assertion can answer it, a judge is a slow, expensive, slightly-wrong version of that.

• Novel failure surfaces. A judge scores what the rubric names. The failure nobody has thought of yet scores fine, which is why scoring never replaces an adversarial pass.

• Genuinely contested quality. When two competent reviewers disagree and stay disagreeing after discussion, that disagreement is real information about the product, not noise for a judge to average over. Take it to the people who own the decision.

• The first two weeks of a feature. Before you know what good looks like, a rubric mostly encodes a guess. Read outputs. Write the rubric second.

The test: for each dimension in your suite, ask what a cheaper check would have to miss to justify the model call. If nothing, you've bought variance at a price.

What to change this week

Don't add dimensions. Calibrate the ones you have.

Pull 50 real outputs, score them by hand against your existing rubric, and compute per-dimension agreement. Budget an afternoon; two people is better than one, and their disagreements are the most useful thing you'll get out of the exercise. Publish the agreement figure next to every eval number you report from then on, because a score without one is an assertion.

Then look at what the misses have in common. If they lean — long, recent, confident, familiar — you've found a bias, and the fix is to replace that dimension with something checkable rather than to tune the judge's wording.

Penhale reports two numbers now: the eval average and the date the judge was last calibrated. Completeness became a four-item factual checklist. The suite got cheaper to run, and the claims assistant's summaries got shorter, which was the outcome the adjusters had been asking for all along.