Distillation takes a large, expensive “teacher” model and uses its outputs to train a smaller “student” model on a narrower task, one the student can then run at a fraction of the cost and latency. The student never has to be as capable as the teacher across every possible query, only on the specific pattern of requests it will actually see in production.
This is what makes it different from simply switching to a smaller off-the-shelf model: a distilled model is trained specifically to match the teacher’s behaviour on the task at hand, so it tends to hold quality far better on that narrow slice than a generic small model would. The trade-off is that it’s an upfront engineering investment, generating training data from the teacher, running the distillation, validating the result, that only pays for itself once query volume is high enough to make the savings matter.
It belongs alongside caching and tiered routing in the standard toolkit for making a production AI workload’s unit economics survive contact with real scale, not something reached for on day one of a pilot.