Self-reported productivity surveys as ROI evidence

For three quarters the firm's coding-assistant ROI figure came from a fortnightly survey: 'How many hours did the assistant save you this fortnight?' The mean answer was 6.2 hours, which multiplied out to a return that made the licence renewal an easy decision. The lab carried the number with a 'signal' tier and a note that nobody had checked it against anything. The Nightingale run was proposed to check it, with a survey arm and a telemetry arm on the same 84 developers.

The telemetry arm measured PR cycle time, review turnaround, rework rate and commits per developer-week, with a baseline from the quarter before licences were issued. Over ten weeks, measured cycle time improved by 9% on the median developer and got worse on the top quartile, who were reviewing more agent-written code. Survey answers over the same weeks did not track any of the telemetry series; the correlation with cycle-time change was 0.11 and the people who reported the most time saved were not the people whose numbers moved. The survey arm was stopped on 14 July and the run continued on telemetry alone.

The refuted claim is that self-report is evidence of ROI. It is evidence of sentiment, which matters for adoption and is not the same thing. r-roi-measure now says to measure from telemetry and cycle time. The renewal decision was re-argued on the telemetry numbers and still went through, on a smaller return and a defensible one.