Dark Harness
code factories
Lights-out software production — a fleet of agents working a backlog with humans only at spec and acceptance — ships real software today on narrow, well-specified backlogs, and fails in ways that are invisible until acceptance on everything else; the reliability gate is not the agents but the absence of a signal between spec and acceptance.
Someone ran it in their own harness. Artifact, no protocol. Decays fast.
Confidence
47%human-committedExpiry
18duntil review · 21 Sep 2026Lead time
—not yet mainstream · opened 5 Jun 2026Ownership
SKSam Kowalczykmonthly cadenceWhere it is
Half a dozen teams have published accounts of running unattended agent fleets against a backlog overnight, and two of the lab's engineers have done it on internal repos. What ships is consistent: small, well-specified, testable tickets get done at a rate no team can match, and the fleet stops being useful exactly where the spec stops being precise. The failure mode is not wrong code; it is plausible code that satisfies the ticket and violates something the ticket did not say. Nobody has an escape-rate figure for unattended production and the vendors selling 'AI software factories' do not publish one. The field is contested because the lab is split on whether the missing piece is better specs (a process problem) or a mid-run signal — an agent that can tell when it is off the rails (a reliability problem). It is also the field the Dark Factory candidate on the next horizon assumes has been solved.
Why a Quantium decision hinges on it
Quantium sells engineering delivery by the pod. A factory that runs overnight on a client backlog changes the unit of sale, the margin, and the liability. The lab needs an answer before a client asks for one in an RFP — which, given the vendor marketing, will be this financial year — and before the firm's own delivery leads try it unsupervised on a client repo, which two already want to.
Field attributes
Position
What is demonstrated, what is hype, what would have to be true.
The shape every position request answers. Signal-tier fields carry a draft; assessed and tested fields carry a validated one.
- 01An overnight fleet of six agents cleared 41 of 48 well-specified tickets on an internal repo with all 41 passing review; the 7 failures were all tickets with an implicit requirement not in the text (tried, two engineers, one repo).
- 02Dependency-upgrade and framework-migration backlogs are the one class where published unattended results exceed 90% acceptance — the spec is the diff between two versions.
- 03Acceptance cost does not fall: reviewing 41 PRs in the morning took the team longer than writing 12 would have, until the fleet was told to write the PR description as an argument for acceptance.
- 01'Software factories' from vendors and from a few loud founders. None publishes an escape rate, a rework rate or an acceptance cost; the demos are greenfield.
- 02'Humans only at spec and acceptance' as a labour saving. In our runs the human hours moved, they did not fall; spec and acceptance are the expensive parts.
- 03The claim that this is the same as agentic SDLC with more agents. It is not — the interactive loop has a human catching drift every few minutes; the dark run has nobody.
- 01A mid-run drift signal — an agent or a check that halts a run when the work has diverged from intent — with a false-negative rate the team would bet a release on. The lab has not seen one.
- 02An escape-rate figure for unattended production on a codebase with a history, from anyone.
- 03Acceptance cost falling below the cost of writing the code, which requires the fleet to produce reviewable evidence, not just diffs.
- 01Preregister a Type 3: same internal repo, three overnight runs, escape rate and acceptance minutes-per-ticket as primaries, tickets stratified by spec precision.
- 02Build and test the cheapest drift signal first — a second agent auditing each ticket against the spec before the PR opens — and measure its false-negative rate.
- 03Tell delivery leads it is not for client repos yet, in writing, and record the rejection so the next proposal sees it.
Signals · 9 in this cluster
What the cluster is made of.
Every item carries its source, tier and sightings. Detector-found signal sits beside human drops; downstream they are indistinguishable except by provenance.
Logged from Claude Code: overnight fleet cleared 41 of 48 tickets; all 7 failures had an unstated requirement
Six agents on a 48-ticket internal backlog, 22:00 to 06:00, no human in the loop. 41 PRs passed review; every failure traced to something the ticket assumed but did not say. Morning review took 5.5 engineer-hours. Two engineers, one repo, tried tier.
extracted claimDark runs succeed on precise tickets and fail on implicit requirements, and human hours move rather than fall.
Logged from Claude Code: spec-audit agent before PR caught 4 of 7 drift failures on the replay
Replayed the seven failed tickets with a second agent auditing each diff against the ticket and the repo's conventions before opening the PR. Caught four; missed three where the constraint was in a different service. Tried tier.
Vendor hiring 'Fleet Reliability Engineer' — 'you will build the signal that stops a bad run'
Argus inference: the vendor selling the factory does not have the drift signal either. The posting describes the missing piece in its own words. Carried as inference.
'Can you quote us autonomous delivery capacity — tickets per night — instead of a pod?'
Line item in a telco RFP, logged by the delivery lead. The firm declined to quote on that basis and offered migration-only scope. The question is the demand signal; the decline is the current position.
Silent Drift: Characterising Specification Violations in Unsupervised Multi-Agent Code Generation
Runs unsupervised agent fleets over 1,200 real issues and classifies failures. 63% of accepted-then-reverted changes passed all tests and violated a constraint present in the codebase but absent from the issue. A spec-audit agent reduced these by 48%.
extracted claimMost dark-run failures pass tests and violate unstated constraints; a spec-audit agent catches about half.
'Code factories are real and they only make one product'
Talk from a former lab researcher arguing dark runs are a solved problem for migrations and an unsolved one for everything else, and that vendors are pricing the first as if it were the second. Demand-band signal with a sceptical framing.
Two vendors launch 'AI software factory' tiers priced per merged PR
Pricing by merged PR rather than by seat — a business-model tell that the vendors expect volume from unattended runs. Neither publishes rework or escape rates. Naming event: 'factory' replaced 'copilot' in both.
'We ran 30 agents overnight for a month. Here is what actually shipped.'
Month-long account from a mid-size SaaS team. Migrations and dependency upgrades shipped at 94% acceptance; feature tickets at 38%. Concludes the backlog, not the fleet, determines the result. Shared widely; three drops in the firm.
extracted claimThe backlog's spec precision, not fleet size, determines dark-run acceptance.
migrate-fleet — unattended framework-migration runner with per-file acceptance
Open-source runner specialised for version-to-version migrations. Star velocity spiked after a large Python 2→3-style framework upgrade was published as a case study. The narrow scope is why it works.
Claims · 4 supporting, 1 refuting
The atoms.
A document cannot go stale; an assertion can. Claims are immutable and stamped with the extractor that produced them, so staleness, diffs and the graveyard operate at claim level.
Migration and dependency-upgrade backlogs are the one production-ready class for dark runs, because the spec is a version diff.
Unattended agent fleets clear well-specified, testable tickets at high acceptance rates; the acceptance rate collapses on tickets with implicit requirements.
The dominant failure mode of dark runs is plausible code that satisfies the ticket and violates an unstated constraint, not code that fails tests.
A second-agent spec audit before PR catches most drift at a cost that keeps the run economical.
Human hours fall under unattended production because humans are only needed at spec and acceptance.
Position history · the diff is the product
3 validation runs against a fixed brief. Confidence 40% → 47%.
Drift signal prototype caught half the failures. Acceptance cost did not fall until the fleet argued for its own PRs. Labour-saving claim downgraded; Type 3 proposed; client repos ruled out in writing.
- The dominant failure mode of dark runs is plausible code that satisfies the ticket and violates an unstated constraint, not code that fails tests.
- A second-agent spec audit before PR catches most drift at a cost that keeps the run economical.
- c-dark-harness-3 ↓ 0.4 → 0.23
Scoring · ordinal bands
Agents propose. A named human commits.
Uncommitted scores are visibly marked and never leave the building. Bands, not point estimates — false precision is the tell that a number was generated rather than derived.
Impact
committed · AWChanges the unit of sale for engineering delivery if it works; a liability if it is tried on a client repo before it does.
Timeline
committed · SKMigration backlogs now; general backlogs after a drift signal exists and is measured.
Cost
agent-estimatedOvernight runs cost $150–400 in tokens each; the Type 3 is three runs plus a pair for three weeks. Agent-estimated.
TAM
agent-estimatedAgent-estimated from the share of AU engineering-services spend that is backlog and migration work. Uncommitted.
Cost of being wrong
committed · LFA plausible-but-wrong change reaching a bank's production is the CPS 230 incident the firm is trying to avoid.
Demand
committed · MLTwo delivery leads want to try it on client repos; one telco RFP asks about 'autonomous delivery capacity'.
Workforce readiness
committed · SKTwo engineers in the lab have run it. Nobody in delivery has written a backlog precise enough for it.
Relevance · per vertical
Why it matters here, or explicitly does not.
Ranking is per vertical, not global. Sector owners commit notes against agent drafts.
Large migration and upgrade backlogs on legacy platforms — the one class where dark runs work — and an RFP that already asks about it.
Mechanism · Migration-only dark runs under a human acceptance gate; general backlog excluded in the SOW.
The economics are attractive and the failure mode is a change-management incident under CPS 230. Not until an escape rate exists.
Mechanism · Would require the drift signal, a measured escape rate, and a change-approval process that treats the fleet as a vendor.
The firm's own internal tooling backlog is the right place to learn this, and the only place it should run this year.
Mechanism · Internal repos, preregistered runs, results into the Type 3.
Red team · the strongest case against
The strongest case against is the lab's own split: if the missing piece is spec precision, this is not a research field, it is a discipline problem, and the answer is the ai-sdlc recommendation already published. If the missing piece is a drift signal, no one has built one that works and the lab's own attempt caught only half of the failures. Either way, 'dark' is the wrong goal — the interactive loop with a human every few minutes is where the evidence is, and the overnight version is a demo optimised for the phrase 'while you sleep'.
- —Human hours did not fall in our own run. The economic thesis rests on a labour saving the tried result contradicts.
- —Every published success is on migrations or greenfield. There is no unattended result on a codebase with a history and a client, and vendors with every incentive to publish one have not.
- —The failure mode — plausible code violating an unstated constraint — is the one that reaches production, because it passes every check the fleet has. The drift signal has to catch what the spec did not say, which is a definition of the thing it cannot do.
Source diversity
- Internal tried findings30%
- Practitioner engineering blogs25%
- ML / SE research20%
- Vendor / Engel25%
A field supported by one epistemic community is a flag, not a finding.
Cross-pollination · typed joins
Connected, not merely similar.
Enabling, compounding, substituting, blocking. A satisfied dependency trigger is a far stronger signal than semantic proximity.
The dark run's acceptance gate is a QA function; if QA is not unattended, the run is not dark.
Spec-first delivery is the discipline that makes tickets precise enough for a fleet; the same recommendation applies with the human removed.
A six-agent fleet on one repo is a fan-out problem; the backpressure patterns are what stopped it thrashing the CI runners.
The next-horizon candidate assumes this field is solved; it is the trigger on that field's dependency.
Share graph
Provenance running forward.
Discovery, not accountability. No counts, no rankings, no rollups to managers.
Convergence · who else is here
- SKSam Kowalczyk · Research engineer · SDLC3 drops
- TOTom Okafor · Research engineer · agents2 drops
- OGOllie Grant · Product engineer2 drops
- MLMarcus Lee · Delivery lead · Telco2 drops
- ?Anonymous · Anonymous drop1 drop
Several people’s drops meet here. An informal working group already exists and probably does not know it.
Lineage
What this field produced, and what it killed.
Experiments, recommendations and graveyard entries stay attached. The reasoning that killed a claim is the reusable asset.
No experiments, recommendations or graveyard entries yet. That is what a candidate looks like.
Open questions · return to the pile
Every run leaves a record. Separately, its question either closes or returns to the pile with notes — which is what the next person proposing the same thing will see.
- 01Is the gap spec precision or a drift signal — and can the Type 3 be designed to tell them apart?
- 02What is the escape rate of a dark run on a codebase with a history, measured by anyone?
- 03At what acceptance-minutes-per-ticket does the economics beat a pod, and does the fleet arguing for its own PRs get there?