Research map / Image generation / Fast diffusion sampling
Generalized Interpolating Discrete Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generalizes masked diffusion to arbitrary interpolating mixing schedules (GIDD), derives its ELBO, and shows that mixing masking with uniform noise enables self-correction in diffusion language models while matching compute-matched MDM perplexity with reweighted training.
Problem
Masked diffusion language models cannot revise already-unmasked tokens, inheriting the irrevocability of autoregressive generation; the design space of discrete diffusion noise processes is under-explored.
Approach
Generalize interpolating discrete diffusion so the forward marginal is a linear mix of data and an arbitrary time-varying mixing distribution, derive closed-form transitions and a continuous-time ELBO, and use a hybrid of masking and uniform noise with a reweighted ELBO plus a self-correction fixed-point sampler.
Key results
- GIDD+ (p_u=0.0) reaches validation PPL 22.29 at 262B tokens vs 23.36 for compute-matched MDM reimplementation on OWT (small models).
- With dynamic weights and weight decay, GIDD+ small validation PPL improves from 28.22 to 24.38 at p_u=0.2 under the ablation.
- Self-correction on p_u=0.2 base model reduces generative PPL from 214 to 93.3 and raises self-accuracy from 62.0% to 73.5%; GPT-4o grades improve by up to 16.6%, while mask-only self-correction degrades quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GIDD+ instead of the MDM reimplementation | DiT small on OWT · OWT validation | PPL: 23.36 -> 22.29 | Helped |
| add uniform noise at p_u=0.1 or p_u=0.2 (ablation) | GIDD+ small · OWT validation | PPL: 23.05 -> 23.67; 23.05 -> 24.38 | Hurt |
| clip the ELBO loss weights (ablation) | GIDD small · OWT validation, p_u=0.0/0.1/0.2 | PPL: 24.36 -> 23.23; 26.88 -> 25.09; 28.22 -> 26.40 | Helped |
| use dynamic loss weights (ablation) | GIDD small · OWT validation, p_u=0.0/0.1/0.2 | PPL: 24.36 -> 23.24; 26.88 -> 23.90; 28.22 -> 24.64 | Helped |
| add weight decay 0.02 (ablation) | GIDD with dynamic loss weights · OWT validation, p_u=0.0/0.1/0.2 | PPL: 23.24 -> 23.05; 23.90 -> 23.67; 24.64 -> 24.38 | Helped |
| add 10% uniform noise (ablation) | GIDD+ · Unconditional generation, 32 denoising steps | generative PPL: 904 -> 387 | Helped |
| apply self-correction (ablation) | GIDD+ base trained with p_u=0.2 · Unconditional generation | generative PPL: 214 -> 93.3 | Helped |
| apply self-correction (ablation) | GIDD+ base trained with p_u=0.2 · Unconditional generation | self-accuracy: 62.0% -> 73.5% | Helped |
| apply self-correction (ablation) | GIDD base trained with p_u=0.0 · GPT-4o sample grading | clarity: 2.51 -> 1.99 (-20.9%) | Hurt |
| apply self-correction (ablation) | GIDD+ base trained with p_u=0.1 · GPT-4o sample grading | clarity: 2.51 -> 2.69 (+7.2%) | Helped |
| apply self-correction (ablation) | GIDD+ base trained with p_u=0.2 · GPT-4o sample grading | clarity: 2.49 -> 2.90 (+16.5%) | Helped |
| add uniform noise to the mask-only model (ablation) | GIDD+ at equal model size and training tokens · Downstream benchmarks | average accuracy: 39.30 -> 38.10; 39.30 -> 36.80 | Hurt |
Surprising findings
- Mask-only MDM shows no self-correction ability and self-correction makes its samples worse, while hybrid-noise models improve.
- Generative perplexity via GPT2-large can be misleadingly low for degenerate, repetitive samples at high min-p cutoffs.
Limitations
- Uniform noise hybrid models lag behind mask-only models in validation perplexity and zero-shot benchmarks.
- Experiments limited to small (up to 320M) models and compute budgets far below frontier LLMs; scaling conclusions are tentative.
- Generative perplexity is an unreliable quality metric and self-correction gains rely on LLM-based grading.
Experimental setup
Datasets: OpenWebText
Benchmarks: OpenWebText validation perplexity ARC-e ARC-c BoolQ Hellaswag PIQA OpenBookQA WinoGrande
Metrics: perplexity generative perplexity unigram entropy self-accuracy zero-shot accuracy
Baselines: GPT2 Llama (retrained) MD4 MDLM MDM (reimpl.)
Builds on: masked diffusion (MDM) MDLM D3PM BERT
Closest papers by meaning
- Simplified and Generalized Masked Diffusion for Discrete Data (NeurIPS 2024 · similarity 0.92)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.91)
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (ICLR 2025 · similarity 0.90)
- The Diffusion Duality (ICML 2025 · similarity 0.89)
- Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.89)
- Remasking Discrete Diffusion Models with Inference-Time Scaling (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.