Research map / Large language models / Language, safety and interpretability
Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows via Gaussian-mixture simulation and SmolLM2 experiments that distilling from a lower-entropy (temperature-scaled) teacher makes the student trade recall for precision.
Problem
Knowledge distillation is widely used for generative models, especially LLMs, but the mechanism by which it improves generative quality is poorly understood.
Approach
Frames KD in generative models as inducing a precision-recall trade-off controlled by teacher entropy via a temperature-like parameter; shows this arises under standard forward-KL distillation in both Gaussian mixtures and LLMs.
Key results
- In the Gaussian simulation, the direct student has precision -20.26 and recall -2.64, while the distilled student (beta=100) has precision -0.70 and recall -42.45.
- In SmolLM2 experiments, student precision rises and recall falls as teacher sampling temperature decreases over tau in {0.8, 0.875, 0.95, 1.0}, averaged over five seeds.
- UMAP projections show distilled students from low-temperature teachers cluster more tightly within the ground-truth embedding region.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill from the selective teacher (β=100) instead of training directly on ground-truth samples | Single-Gaussian student · Gaussian mixture simulation | Precision: -20.26 -> -0.70 | Helped |
| Distill from the selective teacher (β=100) instead of training directly on ground-truth samples | Single-Gaussian student · Gaussian mixture simulation | Recall: -2.64 -> -42.45 | Hurt |
| Lower the teacher sampling temperature | SmolLM2 135M student distilled from SmolLM2 360M · SmolLM2 language modeling | Precision | Helped |
| Lower the teacher sampling temperature | SmolLM2 135M student distilled from SmolLM2 360M · SmolLM2 language modeling | Recall | Hurt |
Surprising findings
- The distilled student from a beta=100 teacher assigns high likelihood to p* samples (precision) but nearly zero probability to many modes, so recall drops sharply.
Limitations
- Experiments only cover pretraining-stage distillation on synthetic data, not instruction tuning or post-training.
- Ground truth is itself a model (SmolLM2 1.7B), and precision/recall are estimated via model likelihoods and embeddings rather than human or task evaluation.
- Toy analysis uses a small Gaussian mixture; the theory is largely qualitative.
Experimental setup
Datasets: 10M sequences sampled from SmolLM2 1.7B (synthetic) 10M sequences per temperature sampled from SmolLM2 360M teacher 100,000 held-out validation sequences
Benchmarks: Gaussian mixture simulation (8-component grid) SmolLM2 precision/recall vs 1.7B ground truth
Metrics: precision (expected log-likelihood under p*) recall (expected log-likelihood of p* samples under student) perplexity UMAP embedding visualization
Baselines: direct student trained on ground-truth samples (p''(D))
Builds on: knowledge distillation SmolLM2 UMAP
Closest papers by meaning
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.92)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.91)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.91)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.90)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.90)
- TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.