Research map / Large language models / Language, safety and interpretability
DistiLLM: Towards Streamlined Distillation for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DistiLLM, a KD framework for auto-regressive LLMs combining a skew KL divergence loss with an adaptive off-policy replay-buffer approach using student-generated outputs, achieving up to 4.3x training speedup over prior KD methods.
Problem
Knowledge distillation for auto-regressive LLMs lacks a standardized objective, and using student-generated outputs to fix training-inference mismatch greatly increases training cost.
Approach
A skew KL divergence (SKL/SRKL) whose gradients and estimator variance are more stable than KLD/RKLD, paired with an adaptive scheduler that raises the probability of using student-generated outputs only when validation loss rises, and a replay buffer with a linearly decreasing replay ratio for off-policy reuse.
Key results
- DistiLLM achieves up to 4.3x training speedup over recent KD methods, and 2.5-4.3x as stated in the paper.
- On SAMSum with T5-XL teacher, DistiLLM reaches 49.11 ROUGE-L for T5-Base vs 48.49 for GKD.
- Off-policy adaptive approach gives 2.2x to 3.4x faster training than on-policy or mixed strategies with minimal performance drop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DistiLLM instead of the strongest listed baseline | T5-Base · SAMSum | ROUGE-L: 48.57 -> 49.11 | Helped |
| use DistiLLM instead of the strongest listed baseline | T5-Small · SAMSum | ROUGE-L: 41.92 -> 42.37 | Helped |
| use DistiLLM instead of the strongest listed baseline | mT5-Base · IWSLT 2017 En-De | BLEU: 30.24 -> 30.32 | Helped |
| use DistiLLM instead of the strongest listed baseline | mT5-Small · IWSLT 2017 En-De | BLEU: 22.04 -> 22.53 | Helped |
| replace on-policy sampling with off-policy sampling (ablation) | DistiLLM · Dolly Eval | score: 26.37 -> 26.12 | Hurt |
| replace on-policy sampling with off-policy sampling (ablation) | DistiLLM · Self-Instruct | score: 13.14 -> 13.16 | Helped |
| replace on-policy sampling with off-policy sampling (ablation) | DistiLLM · Super-Natural | score: 28.24 -> 28.20 | Hurt |
| replace on-policy sampling with off-policy sampling (ablation) | DistiLLM · instruction-following benchmarks | training speed: 2.2× to 3.4× faster | Helped |
| use adaptive off-policy sampling rather than on-policy or mixed sampling (ablation) | DistiLLM · instruction-following benchmarks | performance | No clear effect |
Surprising findings
- Applying the off-policy approach directly to ImitKD and GKD significantly reduces their performance, unlike DistiLLM.
- The validation loss of SKL and SRKL students tends to increase over training, motivating the adaptive SGO scheduler.
Limitations
- Focused on KLD-based objectives; combining with TVD or EMD is left for future work.
- Designed for supervised fine-tuning rather than preference optimization setups.
- Requires teacher and student to share the same tokenizer.
Experimental setup
Datasets: databricks-dolly-15k OpenWebText SAMSum IWSLT 2017 En-De XSum CNN/DM
Benchmarks: Dolly evaluation Self-Instruct Vicuna evaluation Super-Natural Instructions Unnatural Instruction SAMSum IWSLT 2017 En-De
Metrics: ROUGE-L GPT-4 feedback score BLEU training time
Baselines: KD SeqKD ImitKD MiniLLM GKD SFT
Builds on: skew KL divergence MiniLLM GKD ImitKD generalized JSD
Closest papers by meaning
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (ICLR 2024 · similarity 0.94)
- MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.93)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.93)
- DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.92)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.92)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.