One commercial gateway over every model garden
“Standardised on someone else's roadmap.”
Cause
The routing experiment showed the value sits in the routing policy and the cost ledger, both of which we could not get out of the vendor gateway, so the recommendation became a gateway we control.
What was tried, what was hit, what it means
The plan was to pick one of the three commercial gateways, put it in front of Anthropic, Google DeepMind and an Alibaba/Qwen endpoint, and hand delivery teams a single URL. The pitch wrote itself: one bill, one key store, one place for rate limits. We ran a two-month evaluation with the vendor's routing turned on and our own cost ledger alongside.
Three things happened. The vendor's cost-aware routing chose the cheapest model that passed its own quality gate, which was not our quality gate; on the extraction family it routed 38% of calls to a model that failed our eval. The per-call cost data came out as a monthly CSV, which is unusable for the ledger. And the AU data-residency story changed twice in eight weeks, once by a change to their terms and once by a region migration we learned about from a status page.
The question was answered, just not the way the proposal expected. x-gateway-routing validated cost-aware routing across three gardens with our own policy, and r-gateway-default says to route through a gateway you control. Commercial gateways are still fine as the transport underneath. What is dead is standardising on one as the control plane.
Lessons
- 01The routing policy is the product; whoever owns it owns your model choice.
- 02Cost data you cannot query per call is not cost data.
- 03Read a vendor's data-residency terms as a snapshot with a short half-life, not a commitment.
Record
Field
AI Gateway Lineage intact. The reasoning that killed this stays attached to everything it touched.Experiment
Cost-aware routing across three model gardens Kill · Kill if blended cost saving is under 10% by day 12, or if eval pass rate on any task family drops more than 2 points at any saving level.Superseded by
Route through a gateway you control; do not standardise on a vendor'sTry again?
Proposals that touch this entry get its returned notes attached automatically.