Model distillation
Also known as: knowledge distillation
What is Model distillation?
A training method where a large, capable "teacher" model trains a smaller "student" model to copy its behaviour. The student ends up far cheaper and faster to run while keeping most of the quality. Introduced by Hinton and colleagues in 2015, it is how many of the small, efficient models used in production today are made.
Why it matters
Distillation is how a lot of AI becomes affordable in practice: a small model that runs on modest hardware, trained to imitate a large one, can serve the bulk of real requests at a fraction of the cost. The honest caveat is in the words “most of the quality”. The student keeps the teacher’s common behaviour but tends to lose ground on the rare, hard or unusual cases, which is exactly where a cheaper model can let you down if you have not tested those edges. There is a legal wrinkle as well. Distilling from a model you reach through an API may run against that provider’s terms, so where the teacher’s knowledge came from is a question worth asking before it becomes a product.
In practice
A company runs a large, expensive model in production and finds most queries are routine. It distills a small student model that handles those at a fraction of the cost, while routing the rare hard cases to the original. Before trusting the student, it tests specifically on the unusual inputs, where distilled models tend to weaken. The saving is real, but only because the split between easy and hard was measured rather than assumed.
