Research map / Large language models / Language, safety and interpretability
Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Distillation Through Time (SDTT), a distillation method for masked discrete diffusion language models that cuts sampling steps by 32-64x while keeping quality and LAMBADA performance, yielding up to 8x faster decoding than KV-cached GPT-2 at 1.3B scale.
Problem
Autoregressive LLMs generate one token at a time, causing latency; discrete diffusion language models match AR quality but need thousands of denoising steps and cannot use KV caching, so their inference is slow.
Approach
Fine-tune a pretrained MDLM so a student matches the teacher's multi-step sampling outcome in fewer steps, using teacher log-probabilities of tokens denoised over m/k steps as targets (no deterministic DDIM-like mapping needed), applied iteratively over rounds with reverse KL.
Key results
- SDTT reduces decoding steps by 32-64x; the final student achieves lower generative perplexity than GPT-2 with nucleus sampling in 32 steps.
- At 1.3B parameters, diffusion sampling with 16 steps is up to 8x faster than KV-cached GPT-2 on a single A100 (batch size 8).
- KLD-distilled student raises LAMBADA accuracy above the teacher while MCQ accuracy changes minimally (e.g., ARC-Easy 40.57 vs teacher 40.91).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply 7 rounds of SDTT with KLD | MDLM teacher · LAMBADA | accuracy | Helped |
| Use KLD instead of MSE or TVD for SDTT (ablation) | MDLM student · LAMBADA | accuracy | Helped |
| Use KLD instead of MSE or TVD for SDTT (ablation) | MDLM student · conditional generation | MAUVE | Helped |
| Use KLD instead of MSE for SDTT (ablation) | MDLM student · unconditional generation | generative perplexity | Hurt |
| Reduce each SDTT round from 10k to 5k or 2.5k training iterations (ablation) | Iterated SDTT · conditional generation | MAUVE | Hurt |
| Reduce each SDTT round from 10k to 5k or 2.5k training iterations (ablation) | Iterated SDTT · final distillation round | generative perplexity | Helped |
| Distill 4 or 8 teacher sampling steps at once instead of 2 (ablation) | Iterated SDTT · LAMBADA | accuracy | Hurt |
| Use the analytical sampler rather than ancestral sampling to generate teacher targets (ablation) | SDTT · generation | generative perplexity | No clear effect |
| Use the EMA weights as the teacher for successive rounds (ablation) | Iterated SDTT · generation | generative perplexity | Helped |
| Use 4 teacher sampling steps per target and extend rounds to 15k iterations (ablation) | Iterated SDTT · generation | student performance | Hurt |
| Apply SDTT to the pretrained MDLM | MDLM · text generation, 1.3B parameters | speedup: 8x | Helped |
| Apply SDTT to the 169M diffusion model | MDLM · unconditional generation | generative perplexity | Helped |
Surprising findings
- The distilled student's LAMBADA accuracy slightly improves over the teacher's, suggesting better parallel multi-token prediction.
- Using 4 or 8 teacher steps per distillation target hurt performance, and a single model with slowly growing step size diverged.
Limitations
- Diffusion models cannot use KV caching, so latency gains depend on small step counts and the model is not compared with cached AR at all scales beyond 1.3B in main text.
- Distilling more than two teacher steps per round degrades quality; iterated rounds are needed, increasing training cost.
- Generative perplexity is sensitive to precision and evaluated with a proxy model, and models are small and not instruction-tuned.
Experimental setup
Datasets: OpenWebText WebText (MAUVE prompts) LAMBADA
Benchmarks: LAMBADA ARC-Easy ARC-Challenge HellaSwag MathQA PIQA WinoGrande
Metrics: generative perplexity MAUVE self-BLEU LAMBADA accuracy LAMBADA perplexity MCQ accuracy latency
Baselines: GPT-2 MDLM teacher SEDD Llama3 8B (evaluator) MSE distillation TVD distillation
Builds on: MDLM knowledge distillation progressive distillation DDIM
Closest papers by meaning
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.92)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.92)
- Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.91)
- Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.91)
- Energy-Based Diffusion Language Models for Text Generation (ICLR 2025 · similarity 0.91)
- dKV-Cache: The Cache for Diffusion Language Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.