Nightingale measurement run: coding-assistant ROI
ROI- proposed
- voting
- running · 122d
- measuring
- concluded
Preregistration · v2 · 20 Apr 2026 · immutable after start
Hypothesis
Telemetry-derived cycle-time and rework measures on 84 licensed developers show a return on the coding-assistant licence of at least 2× annual cost, and self-reported time saved does not predict which developers' measures moved.
Kill condition
Kill the survey arm if its correlation with any telemetry series is under 0.3 after ten weeks. Kill the run if telemetry attribution covers fewer than 70% of developer-weeks by week six.
Running notes · fed from harness sessions and by the pair
- harness25 Aug 2026SK Sam Kowalczyk
week 16: median cycle time -9%; top quartile +4% (more agent-code review); rework -6%
- manual14 Jul 2026JP Jun Park
Survey arm killed on the kill condition: correlation 0.11 with cycle-time change. Median over-statement 3.4×. Logged as g-roi-survey-method; run continues on telemetry.
- harness18 May 2026JP Jun Park
telemetry pipeline: 84 devs, 4 series, baseline quarter loaded; attribution 91% dev-weeks
- manual20 Apr 2026JP Jun Park
Type 4, signed by Adam with the renewal as the commercial rationale. Predicted: telemetry shows 1.5× not 2×, survey arm dies, confidence 0.6.
Harness notes are auto-captured from Claude Code sessions: model, date, commit, session reference. Never the transcript, code or paths.