Research map / Image generation / Autoregressive and latent generation
Simple and Effective Masked Diffusion Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that simple masked discrete diffusion (MDLM), with a well-engineered training recipe and a substitution-based (SUBS) parameterization yielding a Rao-Blackwellized continuous-time ELBO, reaches new diffusion SOTA on LM1B/OWT and approaches AR perplexity.
Problem
Diffusion language models lag autoregressive models in log-likelihood (perplexity), and prior discrete diffusion results were thought to be weak; the paper asks how much of the gap is due to modeling versus implementation and objective design.
Approach
Restrict discrete diffusion to absorbing-state (masking) processes with a substitution parameterization (zero masking probability and carry-over unmasking), which analytically simplifies the D3PM ELBO into a continuous-time weighted mixture of masked language modeling losses, plus modern engineering (tokenization, DiT backbone, low-discrepancy time sampling) and semi-autoregressive samplers.
Key results
- On LM1B with 33B tokens, MDLM achieves PPL upper bound of 27.04 versus SEDD's 32.79 (about 17% improvement), and 23.00 at 327B tokens vs. AR 20.86.
- On OWT trained for 262B tokens, MDLM reaches PPL upper bound 23.21 versus retrained SEDD 24.10 and AR 17.54.
- Semi-AR generation on 2048-token sequences gives generative PPL 27.18 vs SSD-LM 35.43, with ~25-30x faster decoding (89.3 vs 2473.9 sec/seq).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MDLM instead of SEDD | language modeling · LM1B test | PPL: ≤ 32.79 -> ≤27.04 | Helped |
| Use MDLM instead of SEDD | language modeling · OWT held-out validation | PPL: ≤24.10 -> ≤23.21 | Helped |
| Use MDLM instead of SEDD | OWT-trained language model · Lambada validation zero-shot | PPL: 49.86 -> 47.52 | Helped |
| Use MDLM instead of AR | OWT-trained language model · Lambada validation zero-shot | PPL: 51.28 -> 47.52 | Helped |
| Use MDLM instead of AR | OWT-trained language model · Pubmed zero-shot | PPL: 49.01 -> 41.89 | Helped |
| Use MDLM instead of AR | OWT-trained language model · Arxiv zero-shot | PPL: 41.73 -> 37.37 | Helped |
| Replace SSD-LM SAR decoding with MDLM SAR decoding | semi-autoregressive language generation · Sequences of 2048 tokens | Generative PPL: 35.43 -> 27.18 | Helped |
| Replace SSD-LM SAR decoding with MDLM SAR decoding | semi-autoregressive language generation · Sequences of 2048 tokens | Sec/Seq: 2473.9 -> 89.3 | Helped |
| Fine-tune BERT with MDLM | BERT · GLUE benchmark | Avg: 81.62 -> 82.06 | Helped |
| Use continuous time rather than discrete T=1000 (ablation) | MDLM · LM1B test | PPL: 27.19 ± .07 -> 27.04 ± .01 | Helped |
| Add carry-over unmasking (ablation) | MDLM · LM1B test | PPL: 28.56 ± .15 -> 27.19 ± .07 | Helped |
| Add zero masking probabilities (ablation) | MDLM without carry-over · LM1B test | PPL: 28.56 ± .15 -> 28.51 ± .15 | No clear effect |
Surprising findings
- A re-implemented D3PM with modern engineering nearly matches score-based SEDD, suggesting prior poor D3PM results were due to implementation choices.
- Engineering choices (tokenization, numerically stable objective, architecture) contribute more to performance than the parameterization/objective simplifications.
Limitations
- Results are reported as perplexity upper bounds (ELBO), not exact likelihoods, so comparison with AR exact perplexity is approximate.
- A remaining gap to AR perplexity persists (about 15-25%), and MDLM still underperforms AR on several zero-shot datasets' perplexity overall.
- Semi-AR and sampling are evaluated mainly via generative perplexity under GPT-2, which is an imperfect quality metric.
Experimental setup
Datasets: LM1B OpenWebText C4 HG38 GLUE Genomics Benchmarks PTB Wikitext Lambada AG News Pubmed Arxiv
Benchmarks: LM1B OpenWebText GLUE Genomics Benchmarks HG38 perplexity zero-shot perplexity suite
Metrics: perplexity generative perplexity accuracy F1 seconds per sequence
Baselines: SEDD D3PM (absorb) Diffusion-LM DiffusionBert BERT-Mouth SSD-LM Plaid Transformer autoregressive
Builds on: D3PM SEDD Diffusion-LM MosaicBERT Caduceus
Closest papers by meaning
- Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.94)
- Simplified and Generalized Masked Diffusion for Discrete Data (NeurIPS 2024 · similarity 0.94)
- Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling (ICLR 2025 · similarity 0.93)
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (ICLR 2025 · similarity 0.92)
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.92)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.