SLM / Edge / Tuning
Small models win on narrow, high-volume tasks with a latency or residency constraint, but only after a frontier baseline has been measured on the same eval; fine-tuning without a frontier teacher loses to a prompted frontier model, distillation from one is the only tuning path that has paid back for us, and the economic case erodes every time frontier prices fall.
Experiment run, measured result. The only tier that becomes a recommendation.
Confidence
70%human-committedExpiry
22duntil review · 25 Sep 2026Lead time
+4moahead of mainstream awarenessOwnership
PRPriya Ramanmonthly cadenceWhere it is
The field started as a hypothesis that a fine-tuned 7B would beat frontier on banking classification. It did not — that thesis is in the graveyard — and the field re-formed around what did work: distilling a 3B model from frontier-generated labels for in-store classification, which reached 94% of frontier F1 at a fortieth of the per-call cost and 60 ms on in-store hardware. The pattern is consistent across the literature and the tried-tier findings: fine-tuning on domain data degrades general instruction following, retrieval plus a frontier model wins knowledge tasks, and tuning wins format, latency and residency. The gate is economics because the case keeps moving: frontier per-token prices fell roughly eightfold in a year and each fall shrinks the cost argument, while the residency and latency arguments hold. Small-model releases from Qwen and Mistral have made the model choice a commodity; the eval and the teacher labels are the work.
Why a Quantium decision hinges on it
Two client patterns need it now: in-store vision and classification for retail on hardware that cannot call out, and banking complaints and servicing classification where the data cannot leave the country. In both, the question a delivery team asks is 'retrieval or fine-tuning', and the honest answer depends on a baseline they have not measured. A default that says 'measure frontier first, then distil' saves the six weeks and the wrong answer. It also bears on the open-weight tier and the on-prem field: a small model that pays back is the strongest case for owning inference.
Field attributes
Position
What is demonstrated, what is hype, what would have to be true.
The shape every position request answers. Signal-tier fields carry a draft; assessed and tested fields carry a validated one.
- 01Distilled 3B classifier on in-store hardware: 94.1% of frontier F1, 1/40 the per-call cost, 60 ms local latency (x-slm-edge-classifier).
- 02Fine-tuned 7B on the same task without frontier-generated labels: 81% of frontier F1. The domain-tuning-beats-frontier thesis is refuted on our eval and is in the graveyard.
- 03Retrieval plus a frontier model beat every tuned variant on the knowledge-bearing split of the banking complaints eval; tuning won only the format and latency splits.
- 04A product engineer swapped a 70B extraction call for a tuned 8B on invoice fields at equal accuracy and a twelfth of the cost (tried tier, one harness).
- 01'Fine-tune on your data and beat GPT.' On our eval, without a frontier teacher, it lost by thirteen points.
- 02Small-model release benchmarks. Vendor tables compare against a frontier model from two releases ago.
- 03'Edge AI' as a category. Most enterprise edge cases are a residency or latency constraint wearing a hardware costume; the model choice follows from the constraint.
- 01Frontier per-token prices stop falling, or the residency and latency constraints turn out to be the majority of demand. Currently the cost case has a twelve-month half-life.
- 02A distillation pipeline a delivery team can run — teacher labels, eval, tune, canary — without a research engineer; today it is a lab job.
- 03Small-model tool-use reliability at parity with frontier for the agentic loops that are starting to want on-device inference. Not there.
- 01Ship r-slm-edge as the default: measure frontier on the eval first, distil only when the cost, latency or residency case is explicit.
- 02Keep sa-rag-vs-finetune current on a monthly cadence; it is the most-asked standing answer in the library after cost per Mtok.
- 03Re-run the classifier bench each time frontier pricing moves by more than 2×, because that is when the economics answer changes.
Signals · 10 in this cluster
What the cluster is made of.
Every item carries its source, tier and sightings. Detector-found signal sits beside human drops; downstream they are indistinguishable except by provenance.
Edge SLM for in-store classification: distilled 3B reaches 94% of frontier F1 at 1/40 the cost
Four arms on a Woolworths shelf and product classification eval: prompted frontier, fine-tuned 7B on store labels, distilled 3B on frontier-generated labels, and a stock 3B. The distilled model hit 94.1% of frontier F1 at 60 ms on in-store hardware; the fine-tuned 7B without a teacher hit 81%.
extracted claimDistillation from a frontier teacher, not fine-tuning on domain labels, is the tuning path that reaches frontier-adjacent quality on narrow tasks.
Woolworths Group hiring 'Edge ML Engineer, In-Store Systems' ×3
Three roles for on-device model deployment across the store estate. Inference: the pilot is moving to a programme and the hardware target is settled. Carried as demand, not as a fact about scope.
'Frontier prices fell 8× in a year. Why are you still fine-tuning?'
Charts frontier per-token prices and argues the tuning case is dead for anything but latency and residency. We agree on the cost half and carry it as the reason the gate is economics.
Logged from Claude Code: swapped a 70B extraction call for a tuned 8B on invoice fields; same accuracy, 1/12 the cost
Product engineer tuned an 8B on two thousand invoice examples labelled by the frontier model and swapped it into a live extraction pipeline. Accuracy unchanged on the held-out set, cost down twelvefold. One pipeline, one person; tried tier.
Qwen small-model family: 4B multilingual with tool use, Apache-licensed
Release notes claim parity with a frontier model on 'domain tasks after tuning'; the comparison model is two releases old. Strong base for distillation; the tool-use claim did not hold on our agentic evals. Downloads passed two million inside a month.
extracted claimTuned small models match frontier on domain tasks.
'Can we fine-tune a model on our complaints so nothing leaves the country?'
Asked by a head of customer advocacy. The real question was residency, not cost, and the honest answer was 'you can, and it will lose to a frontier model unless we distil'. Shaped the banking split of the eval and the standing answer.
'Your model, your data, your moat' — vendor keynote on enterprise fine-tuning
Demand-band signal. The keynote sold domain fine-tuning as a general path to beating frontier models; three clients quoted it to us within a month. The claim our bench refuted.
Rationale distillation closes most of the small-model gap on classification
Distilling frontier rationales as well as labels into 1–4B models recovers around 80% of the quality gap on classification and extraction, and almost none of it on multi-step reasoning. Matches the shape of our bench.
Domain fine-tuning degrades instruction following: measurements across 40 tuned checkpoints
Systematic study showing supervised fine-tuning on narrow domain data reduces instruction-following and format compliance in proportion to the domain shift, unless mixed with general data. The mechanism behind the graveyard entry.
Claims · 4 supporting, 1 refuting
The atoms.
A document cannot go stale; an assertion can. Claims are immutable and stamped with the extractor that produced them, so staleness, diffs and the graveyard operate at claim level.
Distillation from frontier-generated labels reaches more than 90% of frontier quality on narrow classification at roughly 1/40 the per-call cost.
Fine-tuning a 7B on domain data without a frontier teacher underperforms a prompted frontier model on the same eval.
Retrieval plus a frontier model beats fine-tuning for knowledge-bearing tasks; fine-tuning wins on format, latency and residency.
Frontier price declines erode the small-model cost case on roughly a twelve-month cycle; the residency and latency cases do not erode.
Fine-tuned small models beat frontier models on domain tasks in general, not just on narrow classification.
Position history · the diff is the product
4 validation runs against a fixed brief. Confidence 45% → 70%.
Edge classifier concluded at 94% of frontier F1. Distillation is the tuning path; retrieval wins knowledge tasks; the cost case erodes with frontier pricing but residency and latency do not. Recommendation and standing answer published.
- Distillation from frontier-generated labels reaches more than 90% of frontier quality on narrow classification at roughly 1/40 the per-call cost.
- Retrieval plus a frontier model beats fine-tuning for knowledge-bearing tasks; fine-tuning wins on format, latency and residency.
- Frontier price declines erode the small-model cost case on roughly a twelve-month cycle; the residency and latency cases do not erode.
Scoring · ordinal bands
Agents propose. A named human commits.
Uncommitted scores are visibly marked and never leave the building. Bands, not point estimates — false precision is the tell that a number was generated rather than derived.
Impact
committed · PRLarge on two patterns, irrelevant to most; the default matters more than the field.
Timeline
committed · PRIn production on one retail pattern already.
TAM
agent-estimatedAgent-estimated from on-device and residency-constrained inference spend. Uncommitted.
Cost
committed · PRTeacher labels are the expense; a bench cycle is two engineers, three weeks, plus frontier spend for labels.
Demand
committed · CDAsked in banking as a residency question, not a cost one. Retail asks about hardware.
Cost of being wrong
agent-estimatedA wrong default costs a delivery team weeks, not a client relationship. Agent-estimated.
Workforce readiness
committed · AWDelivery can run a prompted frontier baseline; the distillation pipeline still needs the lab.
Relevance · per vertical
Why it matters here, or explicitly does not.
Ranking is per vertical, not global. Sector owners commit notes against agent drafts.
In-store hardware cannot call out reliably and the volume makes per-call cost real. The classifier is already in a Woolworths pilot.
Mechanism · Distilled 3B on store devices; frontier for exception review and for the teacher labels.
Complaints and servicing text that cannot leave the country. The case is residency, not cost, and it survives price falls.
Mechanism · Distil on-shore from a frontier teacher run over de-identified text; serve the small model in the bank's tenancy.
On-device clinical classification is attractive and any tuned model touching diagnosis is a TGA software-as-medical-device question.
Mechanism · Depends on whether a distilled classifier for administrative triage stays outside the SaMD boundary. Agent draft.
High-volume intent classification on servicing queues where latency and cost both bind.
Mechanism · Same distillation pattern as retail; teacher labels from the queue transcripts. Agent draft.
Red team · the strongest case against
The strongest case against: we are optimising a line item that is disappearing. Frontier prices fell eightfold in a year, batch and cached inference fell further, and a distillation pipeline is a fixed cost that has to be paid again for every task and every frontier release. The residency argument is the field's real support, and it is an argument for on-prem inference of any size, not for small models. The 94% number is a story about one classification task; the moment the task needs reasoning, the small model is the wrong tool.
- —The cost case had a twelve-month half-life last year and may have a six-month one now. A default that needs re-running every price cut is not a default.
- —One retail classification task and one banking complaints eval. The refutation of domain fine-tuning is on our evals; a different task mix may reverse it.
- —Distillation inherits the teacher's errors and its licence terms. Some frontier providers restrict training on outputs; we have not audited which.
- —Small-model tool-use is unreliable enough that the agentic patterns delivery is actually building cannot use them, which limits the field to classification and extraction.
Source diversity
- ML research30%
- Open-source infra20%
- Vendor20%
- Internal / Engel30%
A field supported by one epistemic community is a flag, not a finding.
Cross-pollination · typed joins
Connected, not merely similar.
Enabling, compounding, substituting, blocking. A satisfied dependency trigger is a far stronger signal than semantic proximity.
Every small model we distil into is open-weight; the parity evals are shared.
A small model that pays back is the strongest case for owning the inference hardware.
Every frontier price cut and caching gain removes part of the reason to tune at all.
On-device recognition and vision are the edge cases with the clearest latency constraint.
Share graph
Provenance running forward.
Discovery, not accountability. No counts, no rankings, no rollups to managers.
Convergence · who else is here
- CDClaire Dubois · Sector owner · Banking1 drop
- MTMei Tanaka · Research lead · evals1 drop
- OGOllie Grant · Product engineer1 drop
- MLMarcus Lee · Delivery lead · Telco1 drop
Several people’s drops meet here. An informal working group already exists and probably does not know it.
Lineage
What this field produced, and what it killed.
Experiments, recommendations and graveyard entries stay attached. The reasoning that killed a claim is the reusable asset.
Distil to a small model only after the frontier baseline is measured on the same eval
strength moderate · 12 citations · review 19 Sep 2026
Retrieval or fine-tuning for this?
strength strong · 36 citations · review 26 Sep 2026
Edge SLM for in-store classification
A 3B model distilled from a measured frontier baseline reaches within 3 points macro-F1 of that baseline on a 22-class shelf-event classification task, running under 150ms on a retail edge box with no network.
Fine-tuned 7B beats frontier on banking classification
“Beat the baseline it was allowed to pick.” · lived 3 months
Open questions · return to the pile
Every run leaves a record. Separately, its question either closes or returns to the pile with notes — which is what the next person proposing the same thing will see.
- 01At what frontier price does the distilled classifier stop paying back, and how close are we?
- 02Which frontier providers' terms permit training on outputs, and does the answer change the teacher choice?
- 03Can small-model tool use reach parity for the narrow agentic loops delivery wants on-device?