cavendish
QueueExperimentsScorecard
Type 3ConvictionValidated

Memory layer bake-off on a 40-session support corpus

Agentic Memory System
  1. proposed
  2. voting
  3. running
  4. measuring
  5. concluded

Preregistration · v2 · 25 Jun 2026 · immutable after start

An extension creates a new version rather than editing the old.

Hypothesis

A structured episodic store with summarised recall beats raw vector recall on decision-consistency precision by at least 10 points at 50k+ tokens of history, at no worse latency.

Kill condition

Kill if episodic+summary trails vector recall on precision by day 14 on the 50k+ split, or if its p50 retrieval latency exceeds 1.5× vector recall at any history length.

MethodFour designs (vector, vector+rerank, graph, episodic+summary) behind one adapter, run against 40 support sessions from a de-identified internal corpus with 220 held-out decision-consistency questions from the LongMemEval-2 split. Precision, p50/p95 retrieval latency and repeat-decision variance at 10k, 50k, 120k tokens of history.
Expected cost$1,400 in tokens, two engineers for three weeks
Expected duration3 weeks

Result

Validated

Validated. Episodic+summary beat vector recall by 19 points precision at 50k+ tokens and ran at 0.6× its p50 latency. Graph memory won only on entity-lookup. Consolidation cut repeat-decision variance from 31% to 8%. Published as r-memory-layer; g-vector-memory-v1 superseded.

Running notes · fed from harness sessions and by the pair

AW
manual · Adam Witanowski · 3 Sep 2026· ⌘↩ to post
  1. manual21 Aug 2026TO Tom Okafor

    Concluded. Latency prediction was wrong in our favour: summarised recall is faster, not slower, because the index is smaller. Write-up to r-memory-layer.

  2. manual12 Aug 2026MT Mei Tanaka

    Repeat-decision variance: 31% re-derived differently without consolidation, 8% with. That is the number Claire will care about.

  3. harness30 Jul 2026TO Tom Okafor

    50k split: episodic 0.83, vector 0.64, vector+rerank 0.72, graph 0.69 precision. episodic p50 latency 0.6x vector. graph wins entity-lookup subset only

  4. harness14 Jul 2026TO Tom Okafor

    adapter: 4 stores wired; corpus loaded; 220 questions; running 10k split

  5. manual25 Jun 2026TO Tom Okafor

    Prereg v2: dropped the 'no-memory' baseline after the vote; it was not informative at 50k. Predicted outcome: episodic wins on precision, loses on latency, confidence 0.6.

Harness notes are auto-captured from Claude Code sessions: model, date, commit, session reference. Never the transcript, code or paths.

Artifacts

repomemory-bench adapter + four storesrepo
evalLongMemEval-2 consistency split, 220 held-outeval
measPrecision × latency × history-length gridmeasurement
noteRepeat-decision variance analysisnotebook