cavendish
RefutedTried

Fine-tuned 7B beats frontier on banking classification

Beat the baseline it was allowed to pick.

21 Oct 202516 Jan 2026 · lived 3 months
Resurrectable — Kekulé re-checks this when the constraint moves

Cause

On a like-for-like eval with a properly prompted frontier baseline, the fine-tuned 7B trailed by 6 F1 points; the earlier 'win' had been against a frontier model with a one-line prompt.

What was tried, what was hit, what it means

An internal deck in October claimed that a LoRA-tuned 7B open-weight model on 12k labelled banking transaction descriptions outperformed a frontier model on a 14-class intent task, at a fortieth of the per-call cost. The number was real. The comparison was not: the frontier baseline had been run zero-shot with a one-line prompt and no examples, and nobody had checked whether that was the strongest reasonable configuration.

A three-day re-run held the eval fixed and gave the frontier model the same class definitions the tuning set encoded, plus twelve examples. The 7B scored 0.86 macro-F1; the frontier baseline moved from 0.81 to 0.92. The cost gap remained real but the accuracy claim reversed. Distillation from the strong baseline, which the original plan had skipped, closed the gap to 2 points and is the path r-slm-edge now describes.

The refuted claim is the narrow one in the title. Small models still win the edge and the cost argument; x-slm-edge-classifier is measuring exactly that on an in-store workload. What is dead is any comparison where the frontier baseline is not measured on the same eval with the same care as the tuned model.

Lessons

  • 01The baseline gets the same effort as the candidate or the comparison is a demo.
  • 02A cost win and an accuracy win are separate claims; keep them on separate lines.
  • 03Distil from the strong model you are trying to beat, not from the labels alone.

Record

OutcomeRefuted
BylineMTMei Tanaka
Born21 Oct 2025
Died16 Jan 2026
Lifespan3 months
Resurrectableyes
TriggerA frontier baseline, prompted with the same class definitions and examples, scoring under 0.88 macro-F1 on the same banking intent eval, or a per-call frontier price above 20× the 7B for the modelled volume.

Try again?

Proposals that touch this entry get its returned notes attached automatically.