Research map / Image generation / Fast diffusion sampling
Encoder-Decoder Diffusion Language Models for Efficient Training and Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes E2D2, an encoder-decoder discrete diffusion language model whose lightweight decoder iteratively denoises while a larger encoder encodes clean tokens, with an efficient block-diffusion training algorithm that halves training FLOPs versus decoder-only BD3LM.
Problem
Decoder-only discrete diffusion language models must run the full network at every denoising step, making sampling expensive; block diffusion training is also costly because clean and noised sequences are both processed by every layer.
Approach
Separate computation into an encoder that represents clean tokens (invoked periodically) and a lightweight decoder that denoises corrupted tokens via cross-attention (invoked at every step), with a vectorized encoder-decoder training scheme using custom block-causal masks for block diffusion.
Key results
- On WMT14 de-en with T=4, E2D2 reaches 24.8 BLEU versus 24.0 for 16-layer BD3LM, with throughput 79.0 vs 47.1 tokens/sec.
- E2D2 trains block diffusion about 40% faster than BD3LM (8.4e4 vs 5.9e4 tokens/sec at batch size 32, L=1024) with comparable perplexity.
- On GSM8K, E2D2 block size S=2 reaches 50.1 pass@1 while S=32 gives 20.9 pass@1 with the highest throughput (62.2 tokens/sec).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use E2D2 block diffusion instead of BD3LM | OWT language modeling · OWT validation | training throughput: 8.4 × 10^4 for E2D2 and 5.9 × 10^4 for BD3LM | Helped |
| Reduce block size from 32 to 2 (ablation) | E2D2 · GSM8K | 0-shot pass@1: 20.9 -> 50.1 | Helped |
| Increase block size from 2 to 32 (ablation) | E2D2 · GSM8K | Tput: 34.8$_{\pm\text{0.4}}$ -> 62.2$_{\pm\text{2.3}}$ | Helped |
| Replace last hidden state with shared KV cache (ablation) | E2D2 trained from scratch · CNN/DailyMail | validation PPL upper bound: 14.94 -> 293.50 | Hurt |
| Replace shared KV cache with last hidden state (ablation) | E2D2 fine-tuned from Qwen3 · GSM8K | validation PPL upper bound: 1.80 -> 1.98 | Hurt |
| Increase diffusion steps per block from 1 to 4 (ablation) | E2D2 · WMT | BLEU: 19.3 -> 24.8 | Helped |
| Increase diffusion steps per block from 1 to 4 (ablation) | E2D2 · WMT | Tput: 116.6$_{\pm\text{3.2}}$ -> 79.0$_{\pm\text{1.5}}$ | Hurt |
| Use E2D2 instead of BD3LM | diffusion models trained on OWT · OWT validation | perplexity: 20.73 -> 21.73 | Hurt |
Surprising findings
- The AR baseline overfits on CNN/DailyMail, while diffusion models are robust to overfitting, so E2D2 outperforms an equal-layer AR model there.
- Shared KV cache variant gave perplexity 293.50 on CNN/DailyMail versus 14.94 for last hidden state, reversing the preference on GSM8K fine-tuning (1.80 vs 1.98).
Limitations
- Larger block sizes improve throughput but reduce generation quality, creating a trade-off.
- Training efficiency and sample quality still lag behind AR models, per the authors.
- Shared KV cache variant underperforms when training from scratch on the larger summarization dataset.
Experimental setup
Datasets: CNN/DailyMail WMT14 de-en GSM8K OpenWebText
Benchmarks: CNN/DailyMail WMT14 de-en GSM8K PTB Wikitext LM1B Lambada AG News Pubmed Arxiv
Metrics: ROUGE BLEU pass@1 perplexity decoding throughput (tokens/sec)
Baselines: AR MDLM BD3LM SEDD
Builds on: block diffusion (BD3LM) MDLM T5 Qwen3
Closest papers by meaning
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (ICLR 2025 · similarity 0.93)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.93)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.92)
- From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.91)
- Remasking Discrete Diffusion Models with Inference-Time Scaling (NeurIPS 2025 · similarity 0.91)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.