Cost redux on tokens
Prompt caching, batching and task-class routing deliver a measured 30–45% per-task cost cut on stable-prefix workloads and nothing like the 60% the blog posts claim; the next factor comes from distillation and context pruning, and list-price falls will beat all of it over twelve months.
Experiment run, measured result. The only tier that becomes a recommendation.
Confidence
83%human-committedExpiry
78duntil review · 20 Nov 2026Lead time
−3moopened after mainstream — recorded honestlyOwnership
PRPriya Ramanmonthly cadenceWhere it is
Opened under pressure from a banking sponsor who wanted a number, and the number now exists. The token cost ledger measured six client patterns end to end: prompt caching saved 31–44% where the prefix was stable and over 2k tokens, went negative once when a prompt reorder dropped the hit rate to 20%, and did nothing on short single-turn calls. Batching halved cost on anything that tolerated hours of latency. Routing added 20–35% on classification and extraction and nothing on agentic loops. Summarised recall was the single largest lever on multi-session agents, which belongs to the memory field. Meanwhile blended list prices for equal capability fell roughly 4× in a year, which is larger than any engineering lever we have and argues for portability over cleverness.
Why a Quantium decision hinges on it
Inference spend is now the largest AI opex line for three of our banking clients and the CFO's office asks 'what does this cost at 10× volume' before approving anything. A standing answer with a measured basis is worth more than a percentage from a vendor deck, and a wrong one — the universal 60% — has already been quoted in one business case we had to walk back. The ledger also tells delivery which levers are safe to promise and which invert under a prompt change.
Field attributes
Position
What is demonstrated, what is hype, what would have to be true.
The shape every position request answers. Signal-tier fields carry a draft; assessed and tested fields carry a validated one.
- 01Across six client patterns, prompt caching saved 31–44% per task on stable prefixes over 2k tokens and 0–6% below that (x-token-cost-ledger).
- 02Batching cut cost 50% on every offline workload in the ledger; no engineering beyond queueing was needed.
- 03A prompt reorder dropped cache hit rate from 78% to 20% and raised cost 25% on one pattern; the saving is fragile in a way nobody documents.
- 04Blended $/Mtok for equal capability fell roughly 4× between September 2025 and August 2026, larger than any lever in the ledger.
- 01'Prompt caching cut our bill 60%.' On one workload with a 20k-token static prefix, maybe. On the six we measured, 31–44% at best.
- 02Semantic caching. Sub-6% hit rates on our workloads; the savings claimed assume duplicate questions that enterprise traffic does not contain.
- 03Token-optimisation SaaS. Every lever we measured is a config change or a queue; paying a margin on top of it is the only way to make it not pay.
- 01Cache pricing staying stable long enough for a business case to survive — two labs changed cache-write pricing in one fortnight in July.
- 02Distillation holding accuracy on a client's real distribution, not our eval; the paper we cite shows the gain evaporating out of distribution.
- 03Context pruning that does not silently drop the one span the answer needed — measured recall loss was 3 points, which some patterns cannot afford.
- 01Keep r-prompt-caching and sa-cost-per-mtok current on a 60-day half-life; the standing answer is asked weekly.
- 02Hand the caching, batching and routing levers to practice as defaults. Keep distillation and pruning open as the next factor.
- 03Add cache hit rate to the gateway's instrument-health metrics so a prompt reorder shows up as a cost alert, not a quarterly surprise.
Signals · 10 in this cluster
What the cluster is made of.
Every item carries its source, tier and sightings. Detector-found signal sits beside human drops; downstream they are indistinguishable except by provenance.
Token cost ledger: six client patterns measured end to end; caching 31–44%, batching 50%, routing 20–35% on narrow classes, one saving inverted
Per-task cost measured through the gateway log across six live patterns before and after each lever. Caching paid on stable prefixes over 2k tokens and not below; batching halved every offline workload; routing helped classification and extraction and not agentic loops; a prompt reorder inverted one caching saving.
extracted claimThe levers are real, bounded and fragile, and none of them is 60%.
$/Mtok tracker: blended cost for equal-capability tier down roughly 4× in twelve months
Tracks list price for the cheapest model clearing a fixed capability bar each month. Down roughly 4× from September 2025 to August 2026. The number that reframes every engineering lever as a timing question.
Two frontier labs change cache-write pricing in the same fortnight
One lab raised the cache-write multiplier and lowered read; the other did the reverse. The ledger's caching numbers moved 4 points in each direction overnight. Evidence that any caching business case needs an expiry date.
extracted claimCache economics change on the provider's schedule, not the client's.
Task-Specific Distillation for Structured Extraction: Cost, Accuracy and Drift
Distils frontier extraction behaviour into an 8B model at one-twelfth the cost with matched accuracy in distribution, then shows a 9-point accuracy drop after a modest input drift. The next cost factor, with a maintenance bill attached.
extracted claimDistillation matches frontier on narrow tasks at a fraction of the cost and loses it under drift.
'Batch APIs: the most underused dollar in your inference bill'
Demand-band signal. A platform engineer's talk on moving offline workloads to batch endpoints; the room's questions suggested most had not. Our ledger's batching number is the same as theirs.
Logged from Claude Code: prompt reorder dropped cache hit rate to 20%, cost up 25%
Product engineer logged from a session: moving the customer context above the system instructions for readability put a dynamic field ahead of the static prefix. Hit rate fell from 78% to 20% and the pattern's cost rose a quarter. Tried tier; reproduced in the ledger.
ctxprune — context pruning for long-prompt LLM calls
Open-source pruner that drops low-attention spans before the call. In the ledger it saved a further 12% of input tokens on the document patterns and cost 3 points of recall. The trade is workload-specific and the repo does not say so.
Analyst note: inference now 60% of enterprise AI operating spend
Demand-band signal. Inference overtook training and platform as the largest AI opex line in the survey. Matches what we see at three banking clients; the figure itself is a survey and carried as such.
'How prompt caching cut our LLM bill by 60%'
Viral engineering post from a company with a 20k-token static system prompt and short user turns — the best possible case for caching. Quoted in one client business case as a general figure. Kept as the strongest version of the claim the ledger refutes.
'What will this cost at ten times the volume?'
Asked by a bank CFO's office before approving a servicing-agent pilot. Delivery answered with a vendor's percentage and the client asked for the basis. Logged unanswered; the ledger is the basis. The question recurs in every business case since.
Claims · 5 supporting, 1 refuting
The atoms.
A document cannot go stale; an assertion can. Claims are immutable and stamped with the extractor that produced them, so staleness, diffs and the graveyard operate at claim level.
Prompt caching yields 30–45% per-task savings on stable prefixes over 2k tokens and near nothing below; it is not a universal 60%.
Batching halves cost on any workload that tolerates hours of latency, with no engineering beyond a queue.
Cache hit rate is fragile to prompt ordering; a reorder that moves a dynamic field above the static prefix can invert the saving.
List-price falls of roughly 4× a year dominate every engineering lever over a twelve-month horizon; optimise for portability across providers before cleverness within one.
Task-specific distillation gives the next cost factor on narrow tasks and loses it under distribution drift; it is a lever with a maintenance bill.
Prompt caching is a universal 60% cost cut.
Position history · the diff is the product
4 validation runs against a fixed brief. Confidence 50% → 83%.
Ledger concluded. List-price falls dominate the levers over twelve months; distillation and pruning are the next factor with a maintenance bill. Recommendation and standing answer published; field converged.
- List-price falls of roughly 4× a year dominate every engineering lever over a twelve-month horizon; optimise for portability across providers before cleverness within one.
- Task-specific distillation gives the next cost factor on narrow tasks and loses it under distribution drift; it is a lever with a maintenance bill.
- c-token-cost-reduction-1 ↑ 0.78 → 0.86
Scoring · ordinal bands
Agents propose. A named human commits.
Uncommitted scores are visibly marked and never leave the building. Bands, not point estimates — false precision is the tell that a number was generated rather than derived.
Impact
committed · PRInference is the largest AI opex line at three banking clients; the levers are worth seven figures a year each.
Timeline
committed · AWMainstream before we opened it. Honest negative lead time.
TAM
agent-estimatedAgent-estimated from global enterprise inference spend. Uncommitted; the addressable share is far smaller.
Cost
committed · PRThe ledger is a gateway query; re-running it is a day.
Demand
committed · CDThe cost-at-10× question is asked in every banking business case; the standing answer is pulled weekly.
Cost of being wrong
agent-estimatedA wrong percentage in a business case is embarrassing and recoverable. Agent-estimated.
Relevance · per vertical
Why it matters here, or explicitly does not.
Ranking is per vertical, not global. Sector owners commit notes against agent drafts.
CFO offices approve inference spend on a forecast; the ledger is the only measured basis we have and the one they ask for.
Mechanism · Ledger numbers feed the business-case template; cache hit rate monitored through the gateway.
High-volume product-description and review-classification workloads where batching alone halves the bill.
Mechanism · Move offline classification to batch endpoints; cache the catalogue prefix.
Contact-centre summarisation at volume, where the prefix is stable and the call transcript is not — the exact shape caching rewards.
Mechanism · Static system prefix cached; per-call transcript uncached; hit rate tracked per queue.
Volumes too low for the levers to matter yet; the interest is in forecasting, not saving.
Mechanism · Would apply if a high-volume document workload lands.
Red team · the strongest case against
The strongest case against: this field is a spreadsheet, not research, and the list-price collapse we ourselves cite makes most of the engineering levers irrelevant within a year. A converged field about config changes is the lab doing delivery's job at a research price.
- —Our own claim 5 says prices fall 4× a year. If true, a 40% caching saving is worth roughly four months of waiting, and the engineering effort to secure it is not obviously positive.
- —Six client patterns is a small ledger. The 31–44% band could be the patterns we chose; a pattern with a short, volatile prefix would have shown 0% and we did not include one.
- —Cache pricing changed twice in July. Every number in the ledger has an expiry the recommendation does not show prominently enough.
- —The largest lever we found — summarised recall — belongs to the memory field. Stripping it out, this field's own contribution is caching plus a queue.
Source diversity
- Model labs / pricing25%
- ML research15%
- Open-source infra15%
- Practitioner / blogs20%
- Internal / Engel25%
A field supported by one epistemic community is a flag, not a finding.
Cross-pollination · typed joins
Connected, not merely similar.
Enabling, compounding, substituting, blocking. A satisfied dependency trigger is a far stronger signal than semantic proximity.
Every lever in the ledger is measured from the gateway log and most are configured in it.
Summarised recall was the largest single lever on multi-session agents; it belongs to that field and shows up in this ledger.
Distillation is the next cost factor and the SLM field owns the method; the ledger owns the measurement.
An ROI number needs a cost denominator with a measured basis; the ledger is that denominator.
Share graph
Provenance running forward.
Discovery, not accountability. No counts, no rankings, no rollups to managers.
Convergence · who else is here
- CDClaire Dubois · Sector owner · Banking1 drop
- ?Anonymous · Anonymous drop1 drop
- MTMei Tanaka · Research lead · evals1 drop
- OGOllie Grant · Product engineer1 drop
Several people’s drops meet here. An informal working group already exists and probably does not know it.
Lineage
What this field produced, and what it killed.
Experiments, recommendations and graveyard entries stay attached. The reasoning that killed a claim is the reusable asset.
Prompt caching: use for stable prefixes over 2k tokens; expect 30–45%, not 60%
strength moderate · 61 citations · review 26 Nov 2026
What does inference actually cost right now?
strength moderate · 48 citations · review 22 Sep 2026
Token cost ledger across six client patterns
Prompt caching delivers at least a 50% reduction in input-token cost on every one of six instrumented client patterns.
Prompt caching as a universal 60% cost cut
“Sixty percent, on the vendor's workload.” · lived 5 months
Open questions · return to the pile
Every run leaves a record. Separately, its question either closes or returns to the pile with notes — which is what the next person proposing the same thing will see.
- 01At what point does the list-price fall make securing a caching saving not worth the engineering hour?
- 02Which of the six patterns would a hostile reviewer say we chose because caching would work on them?
- 03Can cache hit rate be alerted on from the gateway so a prompt reorder is a same-day finding, not a quarter-end one?