Agent-written test suites replace QA engineers
“Could not be wrong, so could not run.”
Cause
Not selected: the hypothesis as written is a headcount claim with no measurable kill condition, and the narrower pilot that could be preregistered (x-agentic-qa-pilot) was selected in its place.
What was tried, what was hit, what it means
The proposal asked for a Type 3 to show that an agent could write and maintain a regression suite well enough that a delivery pod could run without a QA engineer. It was scored in the fortnightly round and came bottom of five, with blind scores averaging 3.4 and a shadow-panel score in agreement. The scorers' notes converged on the same problem: what result would make the proposer stop?
A claim about replacing a role cannot be killed by a test suite metric; the pod would still have a QA engineer at the end of three weeks and the suite would still exist, and both sides could read that as support. The same proposer's fallback, a pilot on a regression-heavy internal codebase with mutation score and escaped-defect count as the measures, was falsifiable and was voted in as x-agentic-qa-pilot in the same round.
This is a rejection of the framing, not the field. Full credit to the proposer; the pilot is theirs and it is in measurement now. If it comes back strong, the headcount question becomes a commercial one for the practice, not a research one for the lab.
Returned notes · what the next proposer sees
Rejected 2026-04-28 because the hypothesis had no kill condition. Wait for x-agentic-qa-pilot to conclude; if mutation score and escaped defects clear the pilot's bar, propose the next step as a delivery question with a measurable outcome (for example, defect rate on a pod running without a QA engineer for a quarter), not as a replacement claim.
The difference between “someone tried this before” and “someone tried this in June, here is what they hit, here is what would need to be true now.”
Lessons
- 01A hypothesis that both outcomes can be read as supporting is not a hypothesis.
- 02Role-replacement claims belong to the practice; the lab tests the capability underneath them.
- 03The fallback proposal was better than the headline one; ask for the fallback first.
Record
Try again?
Proposals that touch this entry get its returned notes attached automatically.