What eval tooling do we use?
The lab's canary suite for release gating, a calibrated LLM judge with a published kappa for scoring, and a per-engagement eval set in the client's repo. No commercial eval platform as of August.
Tier is not strength
Tier says how much we know. Strength says how hard we are telling you to act. Scored independently.
Machine-readable target
{
"configKey": "evals.harness.default"
}Nothing consumes it yet. Day two: findings ship as defaults into the gateway, routing config and skill library.
Body
As of 5 August 2026: three things. The release canary suite (x-eval-canary, v2) gates any model or prompt change on a client pattern. Scoring uses an LLM judge that carries a human-agreement kappa and refuses to publish without one (r-judge-calibration). Every engagement holds its own eval set — 100 to 300 items — in the client repo, versioned with the prompts. We assessed three commercial eval platforms in June and did not adopt one; the gap was calibration, which none of them enforce.
Evidence: x-judge-calibration for the judge, and the canary suite's own record — two regressions caught in July before they reached clients (c-eval-harnesses-2, c-eval-harnesses-4). g-llm-judge-unchecked is why the calibration rule is non-negotiable.
Caveat: the tooling is assessed, not tested, as a whole; the judge is tested. A commercial platform that enforces calibration would change this answer, and one vendor said in August it would ship it. Check before the review date.
What it rests on
Release canaries catch material regressions on structured tasks within hours of a model release; no client pipeline has caught one first.
Public benchmarks saturate within about nine months of release and stop discriminating; task evals on own data are the only durable instrument.
Field
Eval HarnessesExperiment · measuring
Release canary suite v2Experiment · refuted
Judge calibration against human panelGraveyard · refuted
LLM-as-judge without human calibration “Agreed with itself, mostly.”