Research map / Image generation / Fast diffusion sampling
Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows masking diffusion works because it conditions on the known jump schedule, and proposes SCUD, which adds schedule conditioning to any discrete diffusion forward process, achieving state-of-the-art ELBOs on CIFAR-10 images and UniRef50 proteins.
Problem
Masking diffusion consistently outperforms other discrete diffusion models, and it is unclear why; classical discrete diffusion must learn both when and where tokens transition, and structured forward processes have not beaten masking.
Approach
Condition the denoiser on the event schedule (number of events per token) sampled from the known forward process, so the model only learns where to jump. Masking diffusion is recovered as a special case (uniform rate, gamma = 1-1/D), and classical discrete diffusion as gamma to 0.
Key results
- On CIFAR-10 with a structured Gaussian forward process, SCUD achieves state-of-the-art ELBO, beating masking diffusion, with a fraction of the compute of MD4 and tauLDR.
- On UniRef50 proteins, SCUD BLOSUM outperforms masking and classical BLOSUM diffusion, achieving state-of-the-art model fit.
- On LM1B (B=30522), SCUD with a nearest-neighbour graph forward process reaches perplexity 37.63 versus 37.82 for SCUD masking, both far below D3PM (76.9 masking, 149.5 graph).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Schedule conditioning (SCUD) with uniform forward process, sweeping gamma toward 1 (ablation) | uniform discrete diffusion on CIFAR-10 · CIFAR-10 B=128 | model fit (likelihood) | Helped |
| Adding schedule conditioning (gamma toward 1) to structured Gaussian forward process (ablation) | discrete diffusion on CIFAR-10 · CIFAR-10 B=128 | model fit (likelihood) | Helped |
| Schedule conditioning with nearest-neighbour graph forward process | language modeling on LM1B · LM1B | perplexity: SCUD Graph 37.63 vs D3PM Graph 149.5 (different tokenizer caveat) | Helped |
| Adding structure (graph forward process) to SCUD vs masking (ablation) | SCUD on LM1B · LM1B | perplexity: 37.63 vs 37.82 (small gain) | Helped |
| Schedule conditioning applied to BLOSUM structured forward process vs classical BLOSUM | protein model on UniRef50 · UniRef50 | model fit (perplexity) | Helped |
| Classical BLOSUM diffusion vs masking in reimplementation | classical discrete diffusion on UniRef50 · UniRef50 | model fit | Mixed |
| Adding schedule conditioning to Gaussian structured forward process | discrete diffusion on CIFAR-10 · CIFAR-10 | bits per dimension | Helped |
| Structured Gaussian forward process in classical discrete diffusion (without schedule conditioning) | classical discrete diffusion on CIFAR-10 · CIFAR-10 | bits per dimension | Mixed |
Surprising findings
- Masking diffusion's advantage over Gaussian discrete diffusion in classical models is attributed to schedule conditioning rather than the forward process itself.
- In the authors' reimplementation, classical BLOSUM diffusion outperformed classical masking on proteins, contrary to prior reports.
Limitations
- Adding more information to the schedule makes the denoising inference task harder and the third (convergence) term may grow large.
- On language, the structured graph forward process improves perplexity only slightly over masking.
- The paper focuses on likelihood (ELBO) and does not investigate sample quality.
Experimental setup
Datasets: CIFAR-10 UniRef50 One Billion Word (LM1B)
Benchmarks: CIFAR-10 UniRef50 LM1B
Metrics: bits per dimension perplexity ELBO
Baselines: D3PM tauLDR MD4 masking diffusion classical BLOSUM diffusion PixelCNN++ PixelSNAIL
Builds on: D3PM SEDD MD4 tauLDR Gillespie algorithm
Closest papers by meaning
- Simplified and Generalized Masked Diffusion for Discrete Data (NeurIPS 2024 · similarity 0.93)
- Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking (NeurIPS 2025 · similarity 0.91)
- Efficient Denoising Diffusion via Probabilistic Masking (ICML 2024 · similarity 0.90)
- Remasking Discrete Diffusion Models with Inference-Time Scaling (NeurIPS 2025 · similarity 0.89)
- Simple Guidance Mechanisms for Discrete Diffusion Models (ICLR 2025 · similarity 0.89)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.