Research map / Image generation / Fast diffusion sampling
Energy-Based Diffusion Language Models for Text Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EDLM, a residual energy-based model applied over a pretrained masked diffusion LM at each denoising step, with the energy obtained from a pretrained AR model or via NCE fine-tuning, and sampled by parallel importance sampling.
Problem
Discrete diffusion language models underperform autoregressive models, especially with few sampling steps, because their factorized per-token denoising ignores sequence-level dependencies, causing accumulated decoding error.
Approach
Adds a sequence-level residual energy function on top of a pretrained masked diffusion denoiser, p_EDLM(x0|xt) ∝ p_diffusion(x0|xt)·exp(-E(x0,xt)); the energy is set from a pretrained AR model's likelihood (EDLM-AR/coAR) or trained via noise contrastive estimation (EDLM-NCE), and sampled with self-normalized parallel importance sampling.
Key results
- On OpenWebText, EDLM-coAR reaches perplexity 17.58 versus MDLM 23.83 and AR 17.56, and on Text8 reaches BPC 1.24 versus MDLM 1.40.
- With 1024 steps, EDLM-AR gets GPT-2 Gen PPL 25.5 versus MDLM 42.6, and with 256 steps EDLM-NCE gets Llama2 Gen PPL 35.7 versus MDLM 37.2.
- EDLM with importance sampling window 0.2 reaches AR-level Gen PPL (~35.7) in ~13 s versus ~17 s for MDLM, a ~1.3x speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a sequence-level energy model and parallel importance sampling | MDLM · OpenWebText generation, 1024 steps | GPT2 Gen PPL: 42.6 -> 25.5 | Helped |
| add a sequence-level energy model | MDLM · Text8 test | BPC: ≤ 1.40 -> ≤ 1.24 | Helped |
| use carry-over autoregressive energy parameterization (ablation) | EDLM-AR · OpenWebText test | PPL: 20.49 -> 17.58 | Helped |
| use carry-over autoregressive energy parameterization (ablation) | EDLM-AR · Wikitext | PPL: 29.24 -> 28.31 | Helped |
| use carry-over autoregressive energy parameterization (ablation) | EDLM-AR · PTB | PPL: 89.67 -> 89.73 | No clear effect |
| fine-tune the energy function with NCE instead of using the pretrained AR model (ablation) | EDLM · OpenWebText test and zero-shot datasets | PPL: 20.49 vs 21.52; 46.92 vs 49.70 | Mixed |
| apply importance sampling only in the early window, with window length 0.2 (ablation) | EDLM · OpenWebText generation | sampling time: ∼17 seconds -> ∼13 seconds; ∼1.3× | Helped |
| increase the importance-sampling window length (ablation) | EDLM · OpenWebText generation, 1024 steps | GPT2 Gen PPL: 42.6 -> ∼30 | Helped |
| vary importance-sampling sample size below 16 (ablation) | EDLM · OpenWebText generation | generative perplexity | No clear effect |
| use a shorter importance-sampling window rather than a longer one (ablation) | EDLM · OpenWebText generation | sampling efficiency: ∼1.3× | Helped |
Surprising findings
- Carry-over parameterization of the AR energy gives exact ELBO without estimating the partition function, while an importance-sampling window of only 0.2 captures most of the quality gain, suggesting early-stage denoising errors dominate.
Limitations
- Importance sampling adds per-step compute and is memory-bound, so sample sizes above 16 were infeasible on the GPU.
- Perplexities for EDLM-NCE and EDLM-AR are upper bounds because the energy is unnormalized and the partition function is estimated.
- Generation quality is evaluated with Gen PPL from external oracle models, which are imperfect proxies for quality and diversity.
Experimental setup
Datasets: OpenWebText Text8 PTB Wikitext LM1B Lambada AG News Pubmed Arxiv
Benchmarks: Text8 OpenWebText PTB Wikitext LM1B Lambada AG News Pubmed Arxiv
Metrics: bits per character perplexity generative perplexity entropy wall-clock time
Baselines: Transformer AR SEDD MDLM MD4 D3PM SUNDAE Ssd-LM
Builds on: MDLM noise contrastive estimation self-normalized importance sampling SEDD
Closest papers by meaning
- Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.93)
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.93)
- Theoretical Benefit and Limitation of Diffusion Language Model (NeurIPS 2025 · similarity 0.92)
- Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (ICLR 2025 · similarity 0.91)
- The Diffusion Duality (ICML 2025 · similarity 0.91)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.