cavendish
QueueExperimentsScorecard
Type 3QuestionValidated

Token cost ledger across six client patterns

Cost redux on tokens
  1. proposed
  2. voting
  3. running
  4. measuring
  5. concluded

Preregistration · v1 · 9 Feb 2026 · immutable after start

An extension creates a new version rather than editing the old.

Hypothesis

Prompt caching delivers at least a 50% reduction in input-token cost on every one of six instrumented client patterns.

Kill condition

Kill if fewer than four of six patterns show a 50% reduction by day 15, or if the ledger cannot attribute more than 90% of spend to a pattern.

MethodPer-call ledger (tokens in/out, cached in, model, pattern, task) on six de-identified patterns for five weeks: banking servicing, insurance triage, retail range review, two agentic coding loops, one document-extraction batch. Caching on for weeks 3–5; compare against weeks 1–2.
Expected cost$600 in ledger infra, two people for two weeks of setup plus five weeks of collection
Expected duration7 weeks including collection

Result

Validated

Validated as an instrument: the ledger attributed 97% of spend and now feeds sa-cost-per-mtok. The caching hypothesis inside it was refuted (range 4–44%, weighted 26%) and is recorded at g-prompt-caching-universal; r-prompt-caching states the conditions under which 30–45% is realistic.

Running notes · fed from harness sessions and by the pair

AW
manual · Adam Witanowski · 3 Sep 2026· ⌘↩ to post
  1. manual27 Mar 2026PR Priya Raman

    Concluded. Ledger validated as the measurement instrument; caching hypothesis refuted and logged as g-prompt-caching-universal. Proposals corrected.

  2. manual20 Mar 2026JP Jun Park

    Week 5 in. Savings: 41, 44, 19, 12, 7, 4 percent. Weighted 26%. Hypothesis fails on four of six; the ledger itself is the win.

  3. harness24 Feb 2026PR Priya Raman

    ledger live on 6 patterns; attribution 97%; baseline fortnight started

  4. manual9 Feb 2026PR Priya Raman

    Predicted: the 50% holds on three of six, confidence 0.5. Two proposals already carry 60% as a line item; that is the pressure behind this run.

Harness notes are auto-captured from Claude Code sessions: model, date, commit, session reference. Never the transcript, code or paths.

Artifacts

repotoken-ledger (per-call attribution)repo
measSix-pattern caching before/after, five weeksmeasurement
notePrefix stability × call rate vs savingnotebook