Measure ROI from telemetry and cycle time, not surveys
Do not report AI productivity gains from self-reported surveys. Instrument the workflow and measure cycle time, throughput and rework before and after.
Tier is not strength
Tier says how much we know. Strength says how hard we are telling you to act. Scored independently.
Machine-readable target
{
"taskType": "roi-measurement",
"configKey": "nightingale.roi.method"
}Nothing consumes it yet. Day two: findings ship as defaults into the gateway, routing config and skill library.
Body
x-roi-measurement was Nightingale's first measurement run: coding-assistant ROI across four internal teams, with both a survey and telemetry collected in parallel for a quarter. The survey reported a 38% self-assessed productivity gain. Telemetry showed cycle time on comparable tickets down 14% and rework up 6% (c-ai-roi-1). The two numbers are not measuring the same thing, and only one of them is a return.
g-roi-survey-method is the graveyard entry. Survey-based ROI was abandoned as a method, not because people lie but because they report how a tool feels, and a tool that feels fast is not the same as a workflow that finishes sooner. Two published industry studies this year found the same divergence in the same direction (c-ai-roi-2).
What to instrument: ticket-level cycle time, merge-to-deploy time, rework rate (reopened or reverted), and output per person-week, with a comparable baseline from before adoption. It is not exotic. The hard part is that the baseline has to exist, so instrumentation has to start before the rollout, which is the opposite order from how most engagements run (c-ai-roi-3).
Moderate because the run covered coding assistants only. The method transfers to servicing and claims workflows; the specific findings do not.
What it rests on
Self-reported productivity gains from coding assistants overstate telemetry-measured gains by a factor of two to three.
Coding assistants cut PR cycle time but raise review load; the net delivery gain is in the 10–20% band, not 40%.
Vendor acceptance-rate telemetry measures usage, not value; it does not correlate with cycle-time change.
Field
ROIExperiment · running
Nightingale measurement run: coding-assistant ROIGraveyard · refuted
Self-reported productivity surveys as ROI evidence “Everyone said it was working.”