cavendish
QueueExperimentsScorecard
Type 3Observation24d over budget

Spec-first agentic delivery on a live internal repo

AI-SDLC
  1. proposed
  2. voting
  3. running · 45d
  4. measuring
  5. concluded

Preregistration · v1 · 6 Jul 2026 · immutable after start

An extension creates a new version rather than editing the old.

Hypothesis

On the firm's internal engagement-tracker repo, features delivered spec-first (human-reviewed spec, agent-written code, human at merge) ship with at least 40% lower review time per PR and no more post-merge defects than the pod's trailing-quarter baseline.

Kill condition

Kill if review time per PR is not at least 15% lower after 12 features, or if post-merge defects per feature exceed 1.5× baseline at any point.

MethodTwenty features over three weeks. Spec written and reviewed before any code; agent implements against the spec; human reviews the diff against the spec, not the issue. Measure review minutes, defects at 14 days, spec-rewrite count. Baseline from the pod's previous quarter.
Expected cost$1,600 in tokens, two people for three weeks plus the pod's review time
Expected duration3 weeks

Running notes · fed from harness sessions and by the pair

AW
manual · Adam Witanowski · 3 Sep 2026· ⌘↩ to post
  1. manual2 Sep 2026SK Sam Kowalczyk

    Three features to go. Asking for two more weeks rather than concluding at 17; the last three are the largest and will test the drift question.

  2. harness28 Aug 2026SK Sam Kowalczyk

    17 features; review time -41%; 1 defect at 14d (baseline rate would predict 2.3); rewrites 1.4

  3. manual14 Aug 2026TO Tom Okafor

    Overrunning. Features 9–14 stalled: the pod's product owner was on leave and nobody else would sign a spec. Spec-first needs a spec signer; that is a finding in itself.

  4. harness31 Jul 2026SK Sam Kowalczyk

    8 features shipped; review time -47% vs baseline; defects 0; spec rewrites avg 1.6

  5. manual6 Jul 2026SK Sam Kowalczyk

    Predicted 35% lower review time, confidence 0.55. The unknown is how many spec rewrites it takes before the agent stops drifting.

Harness notes are auto-captured from Claude Code sessions: model, date, commit, session reference. Never the transcript, code or paths.

Artifacts

repospec-first templates + agent harnessrepo
measReview minutes and 14-day defects per feature vs baselinemeasurement