Research map / Label-efficient and robust learning / Knowledge distillation
Language Model Distillation: A Temporal Difference Imitation Learning Perspective
Zishun Yu, Shangzhe Li, Xinhua Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply top-p action mask (p=0.8) instead of full vocabulary (ablation) | Bellman Distill · DollyEval, Qwen-2.5 3B->0.5B | Rouge-L: 26.6 -> 27.8 | Improved |
| apply top-p action mask (p=0.8) instead of full vocabulary (ablation) | Bellman Distill · DollyEval, OPT 6.7B->125M | Rouge-L: 23.6 -> 25.1 | Improved |
| lower p from 0.8 to 0.5 (ablation) | Bellman Distill · DollyEval, OPT 6.7B->125M | Rouge-L: 25.1 -> 24.4 | Hurt |
| lower p from 0.8 to 0.5 (ablation) | Bellman Distill · DollyEval, Qwen-2.5 and GPT-2 340M | Rouge-L: 27.8 -> 27.6 (Qwen) | No clear effect |
| add chi2 regularization to distillation objective (ablation) | Bellman Distill · DollyEval, Qwen-2.5 3B->0.5B | Rouge-L: 27.4 -> 27.8 | Improved |
| add chi2 regularization to distillation objective (ablation) | Bellman Distill · DollyEval, GPT-2 1.5B->760M | Rouge-L: 25.9 -> 26.2 | Improved |
| use offline Bellman Distill instead of online MiniLLM | Bellman Distill · Training time, GPT-2 1.5B->340M | training time: 11.2h -> 1.3h | Improved |
| use BD instead of MiniLLM | Bellman Distill · Rouge-L on GPT-2 family | Rouge-L: DollyEval 120M: 24.6 -> 24.7 | No clear effect |
| use BD instead of MiniLLM | Bellman Distill · Rouge-L on OPT and Qwen-2.5 | Rouge-L: OPT-125M Dolly 23.8 -> 25.1; Qwen Dolly 26.7 -> 27.8 | Improved |
| use BD instead of KD/SeqKD | Bellman Distill · DollyEval, OPT-125M | Rouge-L: 21.9 / 22.0 -> 25.1 | Improved |
Closest papers by meaning
- Imitating Language via Scalable Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Adversarial Moment-Matching Distillation of Large Language Models (NeurIPS 2024 · similarity 0.89)
- TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.89)
- On Teacher Hacking in Language Model Distillation (ICML 2025 · similarity 0.89)
- Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback (NeurIPS 2026 · similarity 0.88)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
