cavendish
QueueExperimentsScorecard
Type 3Hypothesis

Agentic QA on a regression-heavy codebase

Fully Agentic QA
  1. proposed
  2. voting
  3. running
  4. measuring · 3d
  5. concluded

Preregistration · v1 · 11 May 2026 · immutable after start

An extension creates a new version rather than editing the old.

Hypothesis

An agent maintaining the regression suite on an internal 60k-line service lifts mutation score from 0.52 to at least 0.75 within three weeks, with no rise in escaped defects during the run.

Kill condition

Kill if mutation score is under 0.6 by day 10, or if any agent-written test is found to assert on its own mocked output rather than behaviour (two instances kills).

MethodAgent runs nightly against the service with a rule that it cannot edit production code or existing tests. Mutation score (Stryker) weekly; escaped defects from the service's incident log; review sample of 40 agent tests by the red team for hollow assertions.
Expected cost$800 in tokens, two people for three weeks
Expected duration3 weeks

Running notes · fed from harness sessions and by the pair

AW
manual · Adam Witanowski · 3 Sep 2026· ⌘↩ to post
  1. harness31 Aug 2026SK Sam Kowalczyk

    week 3: mutation 0.77; escaped defects 0; into measuring

  2. manual24 Aug 2026LF Lena Fischer

    Reviewed 40 tests. One hollow assertion found (asserted a mock returned its own fixture). One instance, not two. Flagged the pattern; agent rule updated.

  3. harness14 Aug 2026SK Sam Kowalczyk

    week 1: mutation 0.52 -> 0.61; 214 tests added; 0 escaped defects

  4. manual11 May 2026SK Sam Kowalczyk

    The fallback that survived the vote after g-agentic-qa-full was rejected. Predicted 0.7, confidence 0.5.

Harness notes are auto-captured from Claude Code sessions: model, date, commit, session reference. Never the transcript, code or paths.

Artifacts

repoqa-agent runner + rulesrepo
measWeekly mutation score and escaped-defect countmeasurement
evalRed-team review of 40 agent-written testseval