Fine-tuned 7B beats frontier on banking classification
“Beat the baseline it was allowed to pick.”
Cause
On a like-for-like eval with a properly prompted frontier baseline, the fine-tuned 7B trailed by 6 F1 points; the earlier 'win' had been against a frontier model with a one-line prompt.
What was tried, what was hit, what it means
An internal deck in October claimed that a LoRA-tuned 7B open-weight model on 12k labelled banking transaction descriptions outperformed a frontier model on a 14-class intent task, at a fortieth of the per-call cost. The number was real. The comparison was not: the frontier baseline had been run zero-shot with a one-line prompt and no examples, and nobody had checked whether that was the strongest reasonable configuration.
A three-day re-run held the eval fixed and gave the frontier model the same class definitions the tuning set encoded, plus twelve examples. The 7B scored 0.86 macro-F1; the frontier baseline moved from 0.81 to 0.92. The cost gap remained real but the accuracy claim reversed. Distillation from the strong baseline, which the original plan had skipped, closed the gap to 2 points and is the path r-slm-edge now describes.
The refuted claim is the narrow one in the title. Small models still win the edge and the cost argument; x-slm-edge-classifier is measuring exactly that on an in-store workload. What is dead is any comparison where the frontier baseline is not measured on the same eval with the same care as the tuned model.
Lessons
- 01The baseline gets the same effort as the candidate or the comparison is a demo.
- 02A cost win and an accuracy win are separate claims; keep them on separate lines.
- 03Distil from the strong model you are trying to beat, not from the labels alone.
Record
Try again?
Proposals that touch this entry get its returned notes attached automatically.