TL;DRDistillation trains a small "student" model to mimic a large "teacher" — compact, cheaper, faster.
Knowledge distillation transfers a big model's behavior into a smaller one: the student trains on the teacher's outputs (soft probabilities, or generated data) rather than raw labels, capturing much of the capability at a fraction of the size and cost. It's how you ship a capable model that fits tighter latency/cost budgets. Distinct from quantization (which shrinks the same model) — distillation creates a new, smaller model.
Key points
Small student mimics a large teacher
Trains on teacher outputs/generated data
Much capability at lower size/cost
Different from quantization (new model)
Common mistakes
Confusing distillation with quantization
Expecting full teacher quality from a tiny student
Poor teacher data → poor student
Try it: Contrast distillation and quantization as compression strategies.
Example code
<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>quantization: same model, fewer bits
distillation: NEW small model mimics big one
both → cheaper inference</pre></body></html>