Multi-agent debate improves reasoning on our tasks
“Adjourned, not decided.”
Cause
A Type 2 run stopped on day 11 when the owner was pulled onto the memory bench and the claims-triage eval it was using was re-baselined under a new extractor, leaving the two completed conditions incomparable with the planned third.
What was tried, what was hit, what it means
The run asked whether three-agent debate (two proposers and a judge, all the same frontier model) beat a single well-prompted agent on the claims-triage eval, at what cost multiple. It was a Type 2, one person, under three days of effort spread over a fortnight. Two of three conditions ran: single agent at 0.74 accuracy, and homogeneous debate at 0.76 for 3.1× the tokens.
The third condition, heterogeneous debate with proposers from different labs, never ran. The owner moved to the memory bench, which had a vote behind it, and in the same week the triage eval was re-baselined under dalton-0.4, which changed the held-out split. Re-running the first two conditions on the new split would have been most of a fresh run, so the attempt was closed and logged.
Two points is inside the noise on a 300-item eval and no claim was made. The interesting condition is the one that did not run. The question is in the pile.
Returned notes · what the next proposer sees
Two of three conditions ran on the old triage split: single agent 0.74, homogeneous debate 0.76 at 3.1× tokens. Not significant and not on the current split. The heterogeneous condition is the one worth running; budget it as a fresh Type 2 on the dalton-0.4 split and do not reuse the January numbers.
The difference between “someone tried this before” and “someone tried this in June, here is what they hit, here is what would need to be true now.”
Lessons
- 01A Type 2 with a single owner stops when the owner does; pair it or accept the risk.
- 02When the eval re-baselines mid-run, the run restarts; note it and move on.
- 03Two points on 300 items is not a finding and should not be written up as a lean either way.
Record
Try again?
Proposals that touch this entry get its returned notes attached automatically.