cavendish
QueueExperimentsScorecard
Type 3PressureValidated

Cost-aware routing across three model gardens

AI Gateway
  1. proposed
  2. voting
  3. running
  4. measuring
  5. concluded

Preregistration · v1 · 4 May 2026 · immutable after start

An extension creates a new version rather than editing the old.

Hypothesis

A routing policy we own, keyed on our per-task eval pass rate, cuts blended cost per task by at least 25% against a fixed frontier default with no drop in eval pass rate.

Kill condition

Kill if blended cost saving is under 10% by day 12, or if eval pass rate on any task family drops more than 2 points at any saving level.

MethodThin gateway (OpenAI-compatible surface) in front of Anthropic, Google DeepMind and a Qwen endpoint. Policy: cheapest model whose trailing-7-day pass rate on the task family clears the family's threshold. Replay 18k production calls from four patterns through fixed-frontier and routed configurations; measure cost and eval pass rate per family.
Expected cost$2,200 in replayed tokens, two engineers for three weeks
Expected duration3 weeks

Result

Validated

Validated. Routed configuration cut blended cost per task by 34% with no family dropping more than 1 point of pass rate. The vendor router comparison failed our extraction eval on 38% of calls. Published as r-gateway-default; g-single-gateway-vendor superseded.

Running notes · fed from harness sessions and by the pair

AW
manual · Adam Witanowski · 3 Sep 2026· ⌘↩ to post
  1. manual30 Jun 2026PR Priya Raman

    Concluded. 34% saving, no family dropped more than 1 point. The per-call ledger is the reusable artefact; r-gateway-default written.

  2. harness18 Jun 2026PR Priya Raman

    vendor gateway comparison arm: their router sent 38% of extraction to a model failing our eval. logged to g-single-gateway-vendor

  3. manual9 Jun 2026SK Sam Kowalczyk

    Replay done. Blended cost -34%. Extraction family routed to Qwen 71% of the time at pass rate 0.93 vs 0.94 frontier. Agentic loops stayed on frontier as expected.

  4. harness21 May 2026PR Priya Raman

    gateway up; 3 gardens; per-call ledger writing to postgres; replay harness 18k calls loaded

  5. manual4 May 2026PR Priya Raman

    Predicted 30% saving, confidence 0.55. The risk is that the extraction family has no cheap model that clears the bar.

Harness notes are auto-captured from Claude Code sessions: model, date, commit, session reference. Never the transcript, code or paths.

Artifacts

repolab-gateway (routing policy + per-call ledger)repo
meas18k-call replay: cost × pass rate by familymeasurement
demoRouting dashboard on the replay setdemo