Research map / Image generation / Fast diffusion sampling
Scaling up Masked Diffusion Models on Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Establishes the first scaling law for masked diffusion language models (MDMs) up to 1.1B parameters, proposes unsupervised classifier-free guidance, and evaluates MDMs against autoregressive models on understanding, generation, reverse curse and temporal shift.
Problem
Scalability and effectiveness of masked diffusion models for core language tasks (understanding, conditional generation) were underexplored, and it was unclear whether MDMs overcome ARM limitations such as the reverse curse and temporal quality degradation.
Approach
Uses the MDM's conditional-on-clean-data property to define unsupervised classifier-free guidance, replacing the unconditional branch with a fully masked condition so CFG works without paired data, and pairs this with IsoFLOP scaling analysis against matched ARMs.
Key results
- The 1.1B MDM outperforms TinyLlama (1.1B, same data) on four of eight zero-shot benchmarks and reaches 58.5 on GSM8K versus 58.6 for Llama-2 7B.
- MDM matches an equal-size ARM's MT-Bench score (1.56 vs 1.57) with 1.4x faster sampling when using ARM KV-cache, while requiring 16x pre-training compute.
- A 1.1B MDM achieves 92% reverse-direction DescriptionToName accuracy versus 0% for Llama-2 13B and GPT-3 175B on the reverse curse dataset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add unsupervised CFG without paired data (ablation) | MDM (220M) · ARC-e, BoolQ, HellaSwag, Obqa, PIQA, RACE, SIQA, and LAMBADA | accuracy: 37.42 -> 39.02; 61.50 -> 62.17; 33.46 -> 34.10; 27.00 -> 34.20; 60.34 -> 60.39; 29.28 -> 30.81; 36.95 -> 37.41; 36.00 -> | Helped |
| Use unsupervised CFG instead of standard CFG (ablation) | MDM conditional generation · MT-Bench | score: 1.53 -> 1.60 | Helped |
| Add CFG rather than sampling without CFG (ablation) | MDM conditional generation · MT-Bench | score: 1.32 -> 1.60 | Helped |
| Increase MDM sampling steps from 128 to 256 | MDM conditional generation · MT-Bench | score: 1.56 -> 1.60 | Helped |
| Use variable-length training data (1%) instead of fixed-length training (ablation) | MDM language understanding · ARC-Easy, BoolQ, HellaSwag, OpenBookQA, PIQA, RACE, SIQA, and LAMBADA | accuracy: 30.13 -> 37.79; 55.29 -> 61.50; 29.16 -> 31.86; 26.20 -> 27.00; 56.04 -> 60.34; 28.52 -> 29.19; 35.21 -> 36.85; 16.51 -> | Helped |
| Pad with mask tokens to the training context length during evaluation (ablation) | MDM language understanding · ARC-Easy, BoolQ, HellaSwag, OpenBookQA, PIQA, RACE, SIQA, and LAMBADA | accuracy: 30.13 -> 38.38; 55.29 -> 59.91; 29.16 -> 31.63; 26.20 -> 27.60; 56.04 -> 60.77; 28.52 -> 28.42; 35.21 -> 37.00; 16.51 -> | Mixed |
| Use chain-rule rather than Monte Carlo likelihood evaluation (ablation) | MDM language understanding · ARC-Easy, BoolQ, HellaSwag, OpenBookQA, PIQA, RACE, and SIQA | accuracy: 39.02 -> 37.88; 62.17 -> 62.17; 34.10 -> 32.20; 30.40 -> 34.20; 59.14 -> 60.39; 30.81 -> 29.67; 37.41 -> 37.10 | Mixed |
| Use unsupervised CFG for reverse queries (ablation) | MDM (1.1B) · DescriptionToName and NameToDescription reverse direction | accuracy: 85 -> 92; 28 -> 37 | Helped |
| Use unsupervised CFG for reverse-query generation (ablation) | MDM (1.1B) · NameToDescription reverse direction | BLEU: 60 -> 67 | Helped |
| Increase MDM sampling steps from 64 to 256 | MDM conditional generation · MT-Bench | sampling time: 204s -> 780s | Hurt |
| Train on SlimPajama and evaluate on newer FineWeb data rather than ARM | MDM (220M) versus ARM (220M) · FineWeb Feb. & Mar. 2024 and Apr. 2024 | perplexity: 27.01 -> 24.06; 26.93 -> 24.01 | Helped |
Surprising findings
- MDMs require about 16 times more pre-training compute than ARMs for comparable validation loss, a constant gap that persists across scales.
- Unsupervised CFG without paired data improves zero-shot accuracy on eight benchmarks (e.g., LAMBADA from 36.00 to 40.99).
Limitations
- MDMs need about 16 times more training compute than ARMs to reach the same loss, and conditional generation still lags in some settings.
- Models only scaled to 1.1B parameters; emergent abilities such as MMLU and ARC-c remain at random-guess level.
Experimental setup
Datasets: SlimPajama ShareGPT augmented GSM8K GSM8K reverse curse fictitious dataset FineWeb
Benchmarks: ARC-e BoolQ Hellaswag OpenBookQA PIQA SIQA RACE LAMBADA GSM8K MT-Bench reverse curse dataset FineWeb perplexity
Metrics: accuracy BLEU perplexity validation loss MT-Bench score sampling time
Baselines: TinyLlama (1.1B) GPT-2 (1.5B) Llama-2 (7B) Llama-2 (13B) GPT-3 (175B) T5 (3B) autoregressive model with KV-cache standard CFG
Builds on: masked diffusion models (MD4/MDLM) classifier-free guidance Chinchilla scaling laws TinyLlama MaskGIT
Closest papers by meaning
- Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling (ICLR 2025 · similarity 0.94)
- Theoretical Benefit and Limitation of Diffusion Language Model (NeurIPS 2025 · similarity 0.94)
- On the Reasoning Abilities of Masked Diffusion Language Models (ICLR 2026 · similarity 0.93)
- Looped Diffusion Language Models (NeurIPS 2026 · similarity 0.93)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.92)
- Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.