LLM-as-judge without human calibration
“Agreed with itself, mostly.”
Cause
Against a five-person human panel on 600 paired items, an uncalibrated frontier judge reached Cohen's κ of 0.41 to 0.58 across three task families, below the 0.8 the hypothesis required and below the 0.6 kill line on two of them.
What was tried, what was hit, what it means
For most of 2025 the eval harness used a frontier model as the judge for anything that was not exact-match, on the assumption that a strong model's judgement tracked a careful human's closely enough to ship on. Nobody had measured the agreement. The calibration run put a human panel of five behind 600 items drawn from extraction, summarisation and agentic-trace grading, and scored the judge against the panel's majority.
Agreement was 0.58 on extraction, 0.49 on summarisation and 0.41 on trace grading. The judge was consistent with itself at 0.87 across re-runs, which is the number that had been mistaken for reliability. On trace grading it systematically rewarded longer traces with more tool calls; the panel systematically penalised them. Two release decisions in Q1 had been made on judge scores that the panel would have reversed.
The hypothesis — an uncalibrated judge agrees with humans — is refuted on our tasks. What replaced it is not 'no judges'. A judge with a per-task rubric and a measured agreement score reached 0.79 to 0.84 on the same items, and r-judge-calibration now requires that every judge ships with that number or does not ship. This entry exists so nobody proposes skipping the panel to save a week.
Lessons
- 01Self-consistency is not agreement with the truth; a judge can be reliably wrong.
- 02Length and tool-call count are the easiest signals for a judge to over-weight and the ones a human panel most discounts.
- 03Every judge score used in a release decision needs a κ next to it.
Record
Field
Eval Harnesses Lineage intact. The reasoning that killed this stays attached to everything it touched.Experiment
Judge calibration against human panel Kill · Kill if κ is below 0.6 on any task family after 200 paired items in that family.Try again?
Proposals that touch this entry get its returned notes attached automatically.