Research map / Image generation / Fast diffusion sampling

Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling

ICLR 2025 · Image generation › Fast diffusion sampling · approach: Discrete diffusion solvers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows masked diffusion models (MDMs) are theoretically time-agnostic masked models, proposes a 20x faster first-hitting sampler, and identifies a float32 Gumbel truncation issue that lowers effective temperature and inflates generative perplexity comparisons vs. ARMs.

Problem

Whether masked diffusion models truly outperform autoregressive models in text generation, and whether MDM training and sampling require continuous time and are efficient; prior evaluations rely on generative perplexity which may be distorted by numerical issues.

Approach

Reformulates the MDM NELBO as a partition over number of masked tokens (time-agnostic), derives a first-hitting sampler that analytically samples unmasking times and only does one categorical draw per step, and shows float32 Gumbel-max sampling truncation distorts categorical probabilities, reducing token diversity and making Gen PPL misleadingly low.

Key results

• With float32 Gumbel sampling, MDM generative perplexity drops below 15 at ~50k steps while entropy keeps decreasing; switching to 64-bit raises Gen PPL to ~126 with entropy ~5.66, and truncation-matched 64-bit gives Gen PPL 28.64.
• The first-hitting sampler is up to 20x faster in wall-clock time than previous MDM sampling strategies, and after fixing precision the true MDM Gen PPL is around 100 versus below 40 for ARMs.
• Theory shows MDM training objective is a time-agnostic ELBO over the number of masked tokens, so the time-independent network is theoretically optimal.

Measured effects reported in the paper

Surprising findings

• Low generative perplexity (<15) for MDMs at high step counts is an artifact of float32 truncated Gumbel sampling that reduces token diversity.
• Caching-based MDM sampling NFE scales with batch size rather than being bounded by sequence length in batched sampling.

Limitations

• Text experiments may inherently favor ARMs since text follows left-to-right order.
• Speedup of 20x may not hold in other settings.
• Experiments use a single pretrained codebase (MDLM/SEDD) on OpenWebText with generative perplexity as main metric.

Experimental setup

Datasets: OpenWebText

Benchmarks: OpenWebText

Metrics: generative perplexity entropy wall-clock time NFE

Baselines: SEDD Absorb MDLM autoregressive model (AR) MDLM with caching strategy

Builds on: MDLM SEDD D3PM DDPM MaskGIT

Closest papers by meaning

• Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.94)
• Theoretical Benefit and Limitation of Diffusion Language Model (NeurIPS 2025 · similarity 0.93)
• Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.93)
• Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking (NeurIPS 2025 · similarity 0.92)
• Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking (NeurIPS 2025 · similarity 0.92)
• Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.92)