Research map / Large language models / Language, safety and interpretability
SPEED: Sharpened-Teacher Distillation for Parallel Decoding of Diffusion Language Models
Qiuhong Shen, Xingyi Yang, Xinyin Ma, Gongfan Fang, Xinchao Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sharpened-teacher distillation (SPEED) to LLaDA | LLaDA-8B-Instruct · GSM8K (5-shot) | decoding steps / speedup: 256 steps -> 26 steps, 9.8× speedup, accuracy 79.3 -> 77.9 | Improved |
| add JSD-based Slow-Fast decoding on top of SPEED | SPEED LLaDA-8B-Instruct · MATH (4-shot) | accuracy: 32.0 -> 32.8 (36 -> 39 steps) | Improved |
| add JSD decoding on top of SPEED | SPEED LLaDA-8B-Instruct · HumanEval (0-shot) | accuracy: 40.9 -> 42.1 | Improved |
| add JSD decoding on top of SPEED | SPEED Dream-7B-Instruct · GSM8K (5-shot) | accuracy: 78.4 -> 78.4 | No clear effect |
| add JSD decoding on top of SPEED | SPEED Dream-7B-Instruct · MBPP (3-shot) | accuracy: 55.6 -> 55.6 | No clear effect |
| add student temperature Ts>1 (ablation) | SPEED distillation · GSM8K (5-shot, LLaDA) | accuracy: 77.6 -> 77.9 (speedup 9.1× -> 9.8×) | Improved |
| add teacher temperature Tt<1 (ablation) | SPEED distillation · GSM8K (5-shot, LLaDA) | speedup: 6.9× -> 9.8× speedup, accuracy 78.1 -> 77.9 | Mixed |
| restrict KL loss to teacher-correct positions instead of all masked tokens (ablation) | SPEED distillation · GSM8K (5-shot, LLaDA) | accuracy: 69.2 -> 77.9 | Improved |
| use selective KL distillation instead of entropy minimization (ablation) | SPEED distillation · GSM8K (5-shot, LLaDA) | accuracy: 76.1 -> 77.9 (31 -> 26 steps) | Improved |
| use training noise level t=0.5 instead of 0.25 (ablation) | SPEED distillation · GSM8K (5-shot, LLaDA) | accuracy: 73.5 -> 77.9 | Improved |
| use training noise level t=0.5 instead of 1.0 (ablation) | SPEED distillation · GSM8K (5-shot, LLaDA) | accuracy: 72.4 -> 77.9 | Improved |
| apply SPEED trained on math data to out-of-domain tasks | SPEED-LLaDA · CNN/DailyMail, TruthfulQA, QASPER | quality / speedup: CNN/DM 19.7 -> 20.8 (3.4×); TruthfulQA 41.8 -> 40.6 (7.1×); QASPER 11.2 -> 10.5 (4.8×) | Mixed |
Closest papers by meaning
- Mean-Field Parallel Decoding for Discrete Diffusion Language Models (NeurIPS 2026 · similarity 0.93)
- Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models (NeurIPS 2026 · similarity 0.92)
- DAWN: Dependency-Aware Fast Inference for Diffusion LLMs (NeurIPS 2026 · similarity 0.91)
- Reconsidering Positional Supervision in Masked Diffusion Language Model Training (NeurIPS 2026 · similarity 0.90)
- Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion (NeurIPS 2026 · similarity 0.90)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
