Prompt caching as a universal 60% cost cut
“Sixty percent, on the vendor's workload.”
Cause
Across six client patterns the measured saving from prompt caching ranged from 4% to 44%, and the 60% figure only appeared on workloads with a stable prefix over 2k tokens and a call rate high enough to keep the cache warm.
What was tried, what was hit, what it means
The 60% figure came from vendor pricing pages and a widely shared post that multiplied the discount on cached input tokens by an assumed cache-hit rate of 90%. It went into two client proposals as a line item before anyone in the lab had measured it. The ledger run was proposed partly to find out whether we had promised something the workloads could not deliver.
The ledger instrumented six patterns for five weeks. Two of them (a banking servicing agent and a retail range-review assistant) had long stable system prompts, high call rates and saved 41% and 44%. Two had short prompts where the cache write cost more than the read saved and came out at 4% and 7%. The remaining two were agentic loops whose prefix changed every few steps and saved 12% and 19%. The weighted average across the six was 26%. The hypothesis was 'at least 50% on every pattern' and it failed on four of six.
The refutation is specific: caching is not universal and 60% is not a planning number. It is still the single cheapest lever on the two patterns where it works, and r-prompt-caching says exactly when to reach for it and what to expect. The proposals were corrected.
Lessons
- 01A discount on a token class is not a saving on a workload until the hit rate is measured on that workload.
- 02Line items that reach a client proposal before a ledger exists are debts, not estimates.
- 03Report the range and the conditions, not the mean; the mean was 26% and useful to nobody.
Record
Field
Cost redux on tokens Lineage intact. The reasoning that killed this stays attached to everything it touched.Experiment
Token cost ledger across six client patterns Kill · Kill if fewer than four of six patterns show a 50% reduction by day 15, or if the ledger cannot attribute more than 90% of spend to a pattern.Try again?
Proposals that touch this entry get its returned notes attached automatically.