Backpressure
Agent systems fail at scale in the fan-out layer before the model layer: unbounded sub-agent spawning and retry storms exhaust rate limits and budgets long before throughput does, and the fix is admission control and a per-task cost circuit-breaker at the orchestrator, not a bigger rate limit.
Experiment run, measured result. The only tier that becomes a recommendation.
Confidence
76%human-committedExpiry
32duntil review · 5 Oct 2026Lead time
—not yet mainstream · opened 2 Apr 2026Ownership
PRPriya Ramanfortnightly cadenceWhere it is
The field opened on an observation from a product engineer's session: a research sub-agent loop spawned 212 children on a malformed spec and hit the organisation's budget cap in eleven minutes. A retail client's weekend pilot cost forty thousand dollars for the same reason a month later. Our experiment reproduced it under a 10× load spike — unbounded fan-out hit provider 429s inside forty seconds and burned 3.2× the budget on retries — and showed that a token-bucket admission controller with a per-task cost cap held p95 latency within 1.6× and spend within 1.1×. The literature on retry amplification agrees, and a telco incident post-mortem dropped anonymously showed 1.1 million retries overnight against a single 429. Providers are moving to token-based priority tiers, which makes client-side admission control more valuable rather than less. The gate is tooling: the pattern is known and cheap, and none of the harnesses delivery uses ship it by default.
Why a Quantium decision hinges on it
Every agentic pattern the firm sells fans out — research loops, document pipelines, multi-step servicing — and every one of them will be run at scale by a client who did not read the rate-limit page. The first production incident on a client agent will be a cost or a rate-limit event, not a wrong answer, and it will land on the delivery team's reputation. A default admission-control layer is the cheapest insurance the lab can issue, and it is also the control CPS 230 tolerances will want to see for any agent in a critical process.
Field attributes
Position
What is demonstrated, what is hype, what would have to be true.
The shape every position request answers. Signal-tier fields carry a draft; assessed and tested fields carry a validated one.
- 01Under a 10× load spike, unbounded fan-out hit provider 429s within 40 s and spent 3.2× the task budget on retries; p95 latency went to 9× baseline before the run was killed (x-backpressure-queue).
- 02Token-bucket admission at the orchestrator plus a per-task cost cap held p95 within 1.6× and spend within 1.1× under the same spike, with 4% of tasks shed to a deferred queue rather than failed.
- 03Org-level budget caps stop all work when one task misbehaves; per-task caps isolate the fault. Measured on the same harness.
- 04The pattern is an afternoon of code against any of the three harnesses delivery uses; none ships it by default.
- 01'Just buy provisioned throughput.' It raises the ceiling; it does not stop a retry storm from reaching it, and it converts a rate-limit incident into a bill.
- 02Vendor 'agent fleet management' products. Most are dashboards over the same unbounded loop.
- 03Rate limits as a procurement conversation. The limit is not the problem; the fan-out that reaches it is.
- 01A harness-native admission-control primitive in at least one of the three harnesses delivery uses, so the pattern is a config key rather than a lab component.
- 02Provider priority tiers stable enough that a client-side controller can plan against them; the current tiers changed twice in three months.
- 03A cost-cap semantic clients accept — shed and defer rather than fail — written into the operational-risk tolerance for the process.
- 01Ship r-backpressure as the default for any agent pattern with fan-out: admission control at the orchestrator, per-task cost cap, deferred queue, before any provider-side change.
- 02Package the controller as a harness adapter for all three harnesses and put it in the delivery template; re-run the spike bench each time a provider changes its rate-limit model.
- 03Take the pattern into the gateway field so the control lives where the traffic already goes.
Signals · 10 in this cluster
What the cluster is made of.
Every item carries its source, tier and sightings. Detector-found signal sits beside human drops; downstream they are indistinguishable except by provenance.
Backpressure bench: unbounded fan-out burns 3.2× budget on retries; admission control holds spend at 1.1×
Three orchestrator configurations under a synthetic 10× load spike on a document-processing pattern: unbounded, org-level cap, and token-bucket admission with per-task cost cap and deferred queue. Unbounded hit 429s inside 40 s; admission control shed 4% of tasks to the deferred queue and held p95 within 1.6×.
extracted claimAgent fan-out fails on rate limits before latency, and orchestrator-level admission control bounds both spend and tail latency.
'Scheduling agent fleets' — infra conference talk
Demand-band signal. 'Fleet', 'scheduler' and 'admission' appeared in five talk titles; the vocabulary is arriving from the systems community. Cross-band ignition with the provider tier change a month earlier.
Anonymised incident post-mortem: overnight batch agent retried 1.1M times against a single 429
A telco delivery lead dropped an excerpt from a client post-mortem, de-identified. A batch enrichment agent with no retry budget looped all night. Nobody noticed until the provider invoice. Sightings across two clients now.
'Stop engineering around rate limits. Buy provisioned throughput.'
Argues that rate limits are a procurement problem and provisioned capacity makes backpressure unnecessary. It raises the ceiling and turns a 429 into an invoice; the retry storm still happens. Kept as the strongest disconfirming voice.
Foundation lab moves to token-based rate limits with priority service tiers
Rate limiting changes from requests per minute to tokens per minute with a priority tier that degrades gracefully under contention. Second change to the model in three months. Client-side controllers that plan against tokens, not requests, gain; everything else has to be rewritten.
extracted claimProvider rate limits are becoming token-based and tiered, which rewards client-side admission control.
'Your agent needs a circuit breaker'
Maps classic resilience patterns — bulkheads, breakers, load shedding — onto agent orchestration. Nothing new to an SRE, and none of it in any agent harness's defaults, which is the post's point and ours.
'The agent pilot cost $40k over a weekend. How do we cap it?'
A merchandising pipeline retried against a rate limit for two days. The sector owner logged the question; the answer at the time was a manual kill switch. Demand signal with a dollar figure attached.
llm-admission — open LLM gateway with token-bucket admission and per-job cost caps
Open-source proxy implementing exactly the pattern, with shed-and-defer semantics. We used it as the reference implementation in the bench; it lacks per-task attribution, which we added.
Logged from Claude Code: research sub-agent loop spawned 212 children on a malformed spec and hit the org budget cap in 11 minutes
A spec with an unterminated list caused a planner to spawn a child per line, recursively. The org-level cap fired and stopped every other agent in the tenant with it. The observation that opened the field and the reason the per-task cap exists.
extracted claimUnbounded sub-agent spawning reaches an org budget cap in minutes, and an org-level cap takes unrelated work down with it.
Retry storms in LLM applications: measuring amplification under rate limiting
Models and measures retry amplification when many clients back off against a shared limit. Finds 2–5× spend amplification with naive exponential backoff and jitter, and that client-side admission removes most of it. Independent of our bench, same numbers.
Claims · 4 supporting, 1 refuting
The atoms.
A document cannot go stale; an assertion can. Claims are immutable and stamped with the extractor that produced them, so staleness, diffs and the graveyard operate at claim level.
Under load spikes, unbounded agent fan-out fails on provider rate limits before it fails on latency, and retries amplify spend two to four times.
Admission control at the orchestrator with a per-task cost cap bounds spend to about 1.1× and holds p95 within 2× under a 10× spike.
Cost circuit-breakers must be per task, not per organisation; org-level caps stop all work when one task misbehaves.
Provider rate limits are moving to token-based priority tiers, which makes client-side admission control more valuable, not less.
Provisioned throughput removes the need for client-side backpressure.
Position history · the diff is the product
4 validation runs against a fixed brief. Confidence 45% → 76%.
Bench concluded: rate limits fail first, retries amplify 3.2×, admission control bounds spend at 1.1×. Recommendation published; gate is tooling because no harness ships the control.
- Under load spikes, unbounded agent fan-out fails on provider rate limits before it fails on latency, and retries amplify spend two to four times.
- Admission control at the orchestrator with a per-task cost cap bounds spend to about 1.1× and holds p95 within 2× under a 10× spike.
- c-backpressure-4 ↑ 0.6 → 0.7
Scoring · ordinal bands
Agents propose. A named human commits.
Uncommitted scores are visibly marked and never leave the building. Bands, not point estimates — false precision is the tell that a number was generated rather than derived.
Impact
committed · PREvery fan-out pattern in delivery; the first production incident on each will be this.
Timeline
committed · PRPattern is done; the work is packaging it into the harness templates.
Cost
committed · PRAn afternoon per harness; the bench is one engineer and a week.
TAM
agent-estimatedAgent-estimated from LLM-gateway and agent-ops tooling spend. Small market, large avoided cost. Uncommitted.
Demand
committed · DSClients ask after the incident, not before. One has, loudly.
Cost of being wrong
agent-estimatedA weekend budget blow-out on a client agent is a reputational event for delivery. Agent-estimated.
Relevance · per vertical
Why it matters here, or explicitly does not.
Ranking is per vertical, not global. Sector owners commit notes against agent drafts.
The forty-thousand-dollar weekend was a retail pilot; the merchandising pipeline fans out over every SKU.
Mechanism · Admission control in the orchestrator, per-task cap sized to the SKU batch, deferred queue for overflow.
Any agent inside a CPS 230 critical process needs a documented tolerance; shed-and-defer is that tolerance made concrete.
Mechanism · Cost and rate tolerances written into the process's operational-risk profile and enforced by the controller. Agent draft.
The dropped incident post-mortem was a telco batch agent; overnight batch is where retry storms go unnoticed until the invoice.
Mechanism · Retry budget per job with exponential backoff and a hard stop; the controller reports shed work to the batch owner.
Agencies run fewer high-fan-out agents today; the pattern matters once document-processing pipelines go agentic.
Mechanism · Same controller; the trigger is the first agentic document pipeline in an agency. Agent draft.
Red team · the strongest case against
The strongest case against: this is a distributed-systems pattern that predates LLMs by twenty years, and the lab has spent a Type 3 rediscovering it. Providers will ship admission control in their SDKs within a release, harness vendors within two, and the field will dissolve into practice having produced a component nobody needed to research. The measured result is real; the question is whether it needed a field.
- —Token buckets and circuit breakers are textbook. The novelty is the cost dimension, and that is a one-line addition.
- —The 10× spike is synthetic. Real client load is bursty in ways the bench did not model, and the 1.1× spend bound may not hold.
- —Provider-side changes are fast. Two rate-limit model changes in three months suggests the target will move before the adapter ships.
- —Shed-and-defer is a policy decision clients may reject; a bank may prefer to fail loudly than defer silently.
Source diversity
- Systems and SRE practitioners30%
- ML and systems research15%
- Vendor20%
- Open-source infra10%
- Internal / Engel25%
A field supported by one epistemic community is a flag, not a finding.
Cross-pollination · typed joins
Connected, not merely similar.
Enabling, compounding, substituting, blocking. A satisfied dependency trigger is a far stronger signal than semantic proximity.
The gateway is where traffic already passes; admission control belongs there once the pattern is stable.
Retry spend is the largest avoidable line in the cost ledger for fan-out patterns.
Agents that spawn agents are undeployable without admission control; this field is the precondition.
Share graph
Provenance running forward.
Discovery, not accountability. No counts, no rankings, no rollups to managers.
Convergence · who else is here
- MLMarcus Lee · Delivery lead · Telco2 drops
- OGOllie Grant · Product engineer1 drop
- DSDev Sharma · Sector owner · Retail1 drop
Several people’s drops meet here. An informal working group already exists and probably does not know it.
Lineage
What this field produced, and what it killed.
Experiments, recommendations and graveyard entries stay attached. The reasoning that killed a claim is the reusable asset.
Put backpressure on agent fan-out before you put it on the model
strength strong · 15 citations · review 22 Jan 2027
Backpressure patterns for agent fan-out
Bounded-concurrency queues with per-parent budgets in front of agent fan-out reduce p95 task latency by at least 30% and cut rate-limit errors by 80% against the unbounded fan-out three patterns run today, at no loss in task completion.
Open questions · return to the pile
Every run leaves a record. Separately, its question either closes or returns to the pile with notes — which is what the next person proposing the same thing will see.
- 01Does the 1.1× spend bound hold on real bursty client load rather than a synthetic spike?
- 02Which harness will ship admission control natively first, and does the adapter become dead code when it does?
- 03Will regulated clients accept shed-and-defer as an operational tolerance, or insist on fail-loud?