Fully Agentic QA
Agents can own test design, execution, triage and regression on a real codebase, but the unit that survives is an agent-run QA function with a human owning the risk model — not a suite that writes itself and a QA team that goes away.
Experiment run, measured result. The only tier that becomes a recommendation.
Confidence
61%human-committedExpiry
21duntil review · 24 Sep 2026Lead time
+4moahead of mainstream awarenessOwnership
SKSam Kowalczykfortnightly cadenceWhere it is
The pilot on a regression-heavy internal repo (x-agentic-qa-pilot) settled the easy half: agents write and maintain test suites faster than people, and they triage flaky failures better than any of our engineers because they do not get bored. The hard half is unsettled. Agent-designed suites over-index on what the code does and under-index on what the business would be embarrassed by, and the coverage number climbs while the escape rate does not fall. Every vendor now sells 'autonomous QA'; nobody publishes escape rates. The field is validating because we have one measured result and it points both ways.
Why a Quantium decision hinges on it
Quantium's delivery model bills QA as a distinct line on most engineering engagements, and three clients have asked whether it should still be there. The answer determines pricing, staffing and what the firm says about its own agentic delivery claims. It also sets the reliability bar for Dark Harness: unattended production is not credible until QA is unattended first.
Field attributes
Position
What is demonstrated, what is hype, what would have to be true.
The shape every position request answers. Signal-tier fields carry a draft; assessed and tested fields carry a validated one.
- 01On a 14k-test internal repo, an agent-run QA loop cut suite maintenance time by 71% and triaged 92% of flaky failures to the correct cause without a human (x-agentic-qa-pilot).
- 02Agent-authored regression tests caught 4 of 5 seeded defects that the human-written suite missed; they missed 2 of 5 that the human suite caught. Complementary, not substitutive.
- 03Escape rate to staging did not move over eight weeks (0.9 → 0.8 per hundred merged PRs), despite line coverage rising from 64% to 88%.
- 01'Coverage went from 60% to 95%.' Coverage is the number agents move most easily and the one least correlated with escapes in our data.
- 02Vendor 'autonomous QA' demos on greenfield repos. Nobody demos on a fifteen-year-old Java monolith with a hand-maintained fixture database.
- 03'QA is dead.' The role changes; the accountability does not, and someone still signs the release.
- 01An agent that can derive a risk model from business context — what would embarrass the client — and weight test design by it. We have not seen one.
- 02Escape rate falling, measured on at least two client codebases, not coverage rising.
- 03A regulator or auditor accepting agent-authored test evidence for a CPS 230 control without a human attestation on each suite.
- 01Re-run the pilot on a client codebase under a Type 3 with escape rate as the only primary metric.
- 02Publish a not-yet recommendation: agents own test maintenance and triage; humans own test design for risk-weighted paths.
- 03Stand up an escape-rate ledger in Nightingale so the firm's own delivery QA becomes measurable before we advise clients on theirs.
Signals · 10 in this cluster
What the cluster is made of.
Every item carries its source, tier and sightings. Detector-found signal sits beside human drops; downstream they are indistinguishable except by provenance.
Agentic QA pilot: 71% less maintenance, escape rate unchanged
Eight weeks of agent-run QA on a 14k-test internal repo with seeded defects. Maintenance and triage effort fell sharply; coverage rose to 88%; escapes to staging stayed flat. Agent and human suites caught different defect classes.
extracted claimAgentic QA cuts maintenance effort by more than half without moving the escape rate.
Big-four bank: 'test engineer' openings down, 'quality engineer, agent systems' up
Argus scan of AU banking engineering postings over six months. Test-engineer titles fell 38%; a new title carrying 'agent' and 'quality' appeared at three of four majors. Inference: the role is being re-titled, not cut.
'The agent tested everything except the thing that would have got me fired'
Widely shared post-mortem: an agent-maintained suite at 91% coverage missed a refund-ordering bug because nothing in the repo said refunds were the risky path. Argues the risk model has to be an input, not an inference.
Panel: 'What does a QA team do in 2027?'
Demand-band signal. Four of five panellists described a smaller team owning risk models and release attestation, with agents underneath. The fifth said the team goes away. Audience vote 70/30 for the first view.
Three test-tooling vendors ship 'autonomous QA agent' in the same fortnight
All three claim end-to-end ownership from test design to sign-off. None publishes escape rates; two demo on greenfield Next.js apps. Naming event: 'autonomous QA' replaced 'AI-assisted testing' across all three.
Analyst note: 'Autonomous testing will remove 40% of QA headcount by 2028'
Headline number with no methodology beyond vendor interviews. Kept as the strongest overstatement in circulation; three clients cited it in the same month.
Coverage Is Not Confidence: LLM-Generated Test Suites and Defect Escape
Evaluates LLM-generated suites on 31 open-source projects with historical defect data. Coverage gains of 20–35 points correspond to escape-rate reductions statistically indistinguishable from zero. Mutation score tracks escapes; coverage does not.
extracted claimLLM-generated coverage gains do not predict defect-escape reductions; mutation score does.
Logged from Claude Code: agent triaged a two-year-old flaky test to a timezone fixture
Product engineer logged from a session: a test that had been quarantined since 2024 was traced by the agent to a fixture with a hard-coded AEDT offset. Fixed in one PR. Tried tier; one instance.
'If your agents write the tests, why is QA still a line on the SOW?'
Asked in a commercial review by a telco client's procurement lead. Logged by the delivery lead; the question that moved the field from candidate to elected under pressure.
SWE-Test-Bench: seeded-defect detection by generated suites
Public harness that seeds defects into real repos and scores generated suites on detection. We ran the pilot repo through it; the agent suite and human suite had a 41% overlap in detected defects.
Claims · 5 supporting, 1 refuting
The atoms.
A document cannot go stale; an assertion can. Claims are immutable and stamped with the extractor that produced them, so staleness, diffs and the graveyard operate at claim level.
Agent-run test maintenance and flaky-triage cut QA effort by more than half on a regression-heavy codebase with no rise in escapes.
Line coverage produced by agents is uncorrelated with escape rate; coverage is the wrong headline metric for agentic QA.
Agent-written and human-written suites catch different defect classes; the union beats either alone by a margin worth paying for.
Agent-designed suites systematically under-test business-risk paths because the risk model is not in the repo.
The QA role is being re-titled, not removed: openings for 'test engineer' are falling while 'quality engineer, agent systems' openings rise.
A fully unattended QA function — design through sign-off — is deployable on enterprise codebases within 18 months.
Position history · the diff is the product
4 validation runs against a fixed brief. Confidence 50% → 61%.
Pilot concluded. Effort saving is real, escape rate is flat, coverage is a vanity metric. The role is re-shaped around the risk model, not removed. Full-replacement thesis sent to the graveyard.
- Agent-run test maintenance and flaky-triage cut QA effort by more than half on a regression-heavy codebase with no rise in escapes.
- Line coverage produced by agents is uncorrelated with escape rate; coverage is the wrong headline metric for agentic QA.
- Agent-designed suites systematically under-test business-risk paths because the risk model is not in the repo.
- Agent-written and human-written suites catch different defect classes; the union beats either alone by a margin worth paying for.
- c-fully-agentic-qa-4 ↓ 0.38 → 0.24
Scoring · ordinal bands
Agents propose. A named human commits.
Uncommitted scores are visibly marked and never leave the building. Bands, not point estimates — false precision is the tell that a number was generated rather than derived.
Impact
committed · AWChanges the staffing model on every engineering engagement the firm runs.
Timeline
committed · SKMaintenance and triage are usable now; design and sign-off are not.
TAM
agent-estimatedAgent-estimated from global QA outsourcing spend and test-tooling revenue. Uncommitted.
Cost
committed · SKOne engineer on the pilot repo for six weeks; tokens under $2k.
Cost of being wrong
committed · LFIf we tell a bank QA is unattended and it is not, the escape lands in production under CPS 230.
Demand
committed · MLThree engagements asked whether QA should remain a billed line; one RFP scores 'AI-driven testing' explicitly.
Workforce readiness
agent-estimatedDelivery pods can run maintenance/triage today; nobody outside the lab has written a risk-weighted test plan for an agent. Agent-estimated.
Relevance · per vertical
Why it matters here, or explicitly does not.
Ranking is per vertical, not global. Sector owners commit notes against agent drafts.
CPS 230 makes test evidence a control artefact. Agent-authored evidence is only useful if an auditor will accept it, which none has yet.
Mechanism · Agents maintain and run suites; a named engineer attests each release against a risk-weighted test plan.
Provisioning and billing regression suites are large, brittle and mostly maintenance. This is where the 71% lands first.
Mechanism · Agent-run maintenance and flaky triage over the existing suite before any redesign.
Procurement panels still specify tester headcount. Until the buying template changes, the saving is not bankable.
Mechanism · Would need DTA guidance on agent-produced assurance evidence.
Promotional-pricing logic breaks in ways the code never predicts. Agents wrote 400 tests and none exercised a stacked-discount path.
Mechanism · Human-authored risk model feeds the agent's test design; agents own everything below it.
Red team · the strongest case against
The strongest case against: we measured the easy metrics on a codebase we already understood. Maintenance and triage were always the cheap part of QA; the expensive part is knowing what to test, and the pilot showed agents do not know that. A 71% saving on 30% of the work is a 21% saving being sold as a transformation.
- —Escape rate did not move. On the only metric that matters to a client, eight weeks of agentic QA produced no measurable change.
- —The pilot repo has a complete, honest test suite already. Most client codebases do not, and an agent maintaining a bad suite maintains it faster.
- —The 'union beats either' result depends on keeping the human suite, which means keeping the humans. The economics disappear if the headline is staffing.
Source diversity
- Software engineering research30%
- Practitioner posts25%
- Vendor / analyst20%
- Internal / Engel25%
A field supported by one epistemic community is a flag, not a finding.
Cross-pollination · typed joins
Connected, not merely similar.
Enabling, compounding, substituting, blocking. A satisfied dependency trigger is a far stronger signal than semantic proximity.
Spec-first delivery produces the acceptance criteria an agentic QA loop needs as its risk model.
Lights-out production is not credible until the QA function is unattended and the escape rate says so.
Judge calibration methods transfer directly to deciding whether an agent-written test asserts anything.
Escape rate and suite maintenance time are the two QA metrics Nightingale can already read from telemetry.
Share graph
Provenance running forward.
Discovery, not accountability. No counts, no rankings, no rollups to managers.
Convergence · who else is here
- MTMei Tanaka · Research lead · evals2 drops
- SKSam Kowalczyk · Research engineer · SDLC2 drops
- OGOllie Grant · Product engineer1 drop
- MLMarcus Lee · Delivery lead · Telco1 drop
- ?Anonymous · Anonymous drop1 drop
- LFLena Fischer · Red team & assurance1 drop
- RMRohan Mehta · Exec sponsor1 drop
Several people’s drops meet here. An informal working group already exists and probably does not know it.
Lineage
What this field produced, and what it killed.
Experiments, recommendations and graveyard entries stay attached. The reasoning that killed a claim is the reusable asset.
Agentic QA on a regression-heavy codebase
An agent maintaining the regression suite on an internal 60k-line service lifts mutation score from 0.52 to at least 0.75 within three weeks, with no rise in escaped defects during the run.
Agent-written test suites replace QA engineers
“Could not be wrong, so could not run.” · lived 14 days
Open questions · return to the pile
Every run leaves a record. Separately, its question either closes or returns to the pile with notes — which is what the next person proposing the same thing will see.
- 01Can a risk model be expressed in a form an agent can weight test design by, and who in a delivery pod writes it?
- 02Does the escape-rate result hold on a client codebase without a pre-existing honest suite?
- 03Will an APRA-regulated entity's auditor accept agent-authored test evidence, and under what attestation?