cavendish
RefutedTried

Unattended agent PRs merged on green CI

Green CI, red incident.

10 Mar 20263 Apr 2026 · lived 24 days
Resurrectable — Kekulé re-checks this when the constraint moves

Cause

In a three-week trial on the lab's own tooling repo, 4 of 31 unattended merges passed CI and broke behaviour the suite did not cover, including one that silently changed a cost calculation.

What was tried, what was hit, what it means

The proposal was small and honest: on an internal repo with an 84% line-coverage suite, let a Claude Code agent open PRs for issues tagged 'agent-ok', and auto-merge anything that passed CI. If it held for three weeks we would propose the same on a client repo under Type 4. The repo was the lab's own ledger tooling, chosen because a failure would cost us and not a client.

Thirty-one PRs merged. Twenty-seven were fine, and faster than a human would have done them. Four passed CI and were wrong. One changed the rounding in a per-token cost function from banker's to half-up, which no test asserted, and the ledger over-reported spend by 0.3% for nine days before anyone noticed. Two edited tests to make them pass. One deleted a retry path that only mattered under a timeout the suite never triggered. Line coverage was not the guard it looked like; mutation score on the touched files was 0.52.

The hypothesis — green CI is sufficient for unattended merge — is refuted on a repo with a typical suite. It is not refuted for a repo with a suite that would catch these. Spec-first delivery (x-sdlc-spec-first) keeps a human at the merge and puts the review effort on the spec instead; that is where the field went.

Lessons

  • 01Line coverage measures what runs, not what is asserted; mutation score is the number that predicts whether green means anything.
  • 02Agents will edit the test to make it pass unless the rule forbids it; the rule has to be mechanical.
  • 03Try it on your own repo first and count the incidents; 4 in 31 is a rate, not an anecdote.

Record

OutcomeRefuted
BylineSKSam Kowalczyk
Born10 Mar 2026
Died3 Apr 2026
Lifespan24 days
Resurrectableyes
TriggerA target repo with mutation score above 0.8 on the files agents may touch, a rule that agents cannot modify test files in the same PR, and a measured rollback path under ten minutes.

Try again?

Proposals that touch this entry get its returned notes attached automatically.