Every LLM judge ships with a human agreement score or does not ship

x-judge-calibration put four judge configurations against a 300-item human panel on three of our task types. Uncalibrated judges agreed with humans between 0.52 and 0.71 kappa; the same judges with a rubric tuned on 60 panel items reached 0.81–0.88 (c-eval-harnesses-1, c-eval-harnesses-2). The uncalibrated numbers were not noise: two judges were systematically lenient on fluent wrong answers, which is the failure mode that matters most for client-facing agents.

The graveyard entry g-llm-judge-unchecked records what it cost us. A release canary in March reported a 6-point improvement on claims summarisation that the human panel later reversed. The judge had learned to reward length. The improvement was reported to a client before it was unreported.

The rule is a 'do not' because the alternative — a judge score with no agreement figure attached — is worse than no score. It looks like measurement. Every judge in the eval harness now carries its kappa, the panel size, and the date of the last calibration, and the canary suite refuses to publish without them (x-eval-canary, c-eval-harnesses-3).

Cost: sixty panel items per task type, roughly a day of two people's time. Recalibrate on model change; a judge calibrated on one model version drifted by 0.09 kappa on the next.