Overview
Distillation is the most reliable lever for cutting inference cost: train a small model to imitate a large one on the distribution you care about, and serve the small one. But naive distillation fails in a specific, predictable way — and knowing the failure mode is most of the guide.
What breaks first is agentic capability. Fluency and single-turn knowledge transfer well; multi-step tool use degrades fast, because tool-call tokens are rare in training mixes and errors compound across steps. A student one-tenth the size can match the teacher's prose while failing half its tool workflows.
The countermeasures are unglamorous. Select trajectories, not samples — distill on complete multi-step episodes that succeeded, so the student sees recovery and sequencing, not just answers. And weight the loss toward decision points: the tokens that pick a tool and bind its arguments matter more than the prose between them.
Evaluate on workflows, not benchmarks. Static QA scores will tell you the distillation worked; production will tell you it didn't. Our coordination benchmark suite exists partly because nothing off the shelf measured the thing that breaks.
When should you distill at all? When your workload is narrow enough to define a distribution, stable enough to justify the training run, and large enough that serving costs dominate. Two out of three, fine-tune the small model directly. One out of three, keep paying for the big one — it's cheaper than the alternative.
The full method — trajectory selection, tool-call-weighted loss, results across three model families — is published in our research section.
