Research map / Image generation / Fast diffusion sampling

Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models

ICLR 2025 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Block Discrete Denoising Diffusion Language Models (BD3-LMs), which autoregressively model blocks of tokens with discrete masked diffusion inside each block, plus an efficient training algorithm and variance-minimizing clipped noise schedules to narrow the perplexity gap to AR models.

Problem

Discrete diffusion language models are limited to fixed-length generation, cannot use KV caching, and lag autoregressive models in perplexity, partly due to high gradient variance during training.

Approach

Define an autoregressive distribution over blocks of tokens with discrete masked diffusion within each block, train all blocks in one vectorized forward pass using a block-causal attention mask on concatenated noisy and clean sequences, and learn data-driven clipped masking-rate schedules that minimize gradient variance.

Key results

• On OpenWebText, BD3-LM with block size 4 reaches perplexity at most 20.73 versus 22.98 for MDLM and 24.10 for SEDD among diffusion models (AR 17.54).
• On LM1B with clipped schedules, BD3-LM at L'=4 reaches perplexity 29.16 versus 31.78 for MDLM, an up-to-13% improvement.
• Generated samples reach up to ~9982 tokens (block size 16) versus SEDD's 1024 training-context cap; gen. PPL at L=1024 is 25.7 for L'=4 vs 46.8 for MDLM.

Measured effects reported in the paper

Surprising findings

• A block-size-1 diffusion model trained with the standard masking schedule has a two-point perplexity gap to AR, which is attributed to high gradient variance rather than the objective itself.
• Training with full masking (q(x_t = m | x) = 1) for L'=1 makes the diffusion objective match the AR objective and closes the gap (22.88 PPL).

Limitations

• Training is more expensive than standard diffusion training; the vectorized algorithm keeps overhead under 2x, and pretraining with a standard diffusion loss is used to reduce the gap.
• Blocks are generated sequentially, so small block sizes inherit AR-like speed and controllability constraints.
• Optimal block size is task-specific.

Experimental setup

Datasets: LM1B OpenWebText

Benchmarks: LM1B OpenWebText PTB Wikitext Lambada AG News Pubmed Arxiv

Metrics: perplexity variance of NELBO generative perplexity NFEs median and max generated length

Baselines: AR Transformer SEDD MDLM D3PM (absorb) SSD-LM Transformer-X Base

Builds on: D3PM MDLM SEDD SSD-LM

Closest papers by meaning

• Encoder-Decoder Diffusion Language Models for Efficient Training and Inference (NeurIPS 2025 · similarity 0.93)
• Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.92)
• Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.92)
• Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.92)
• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.91)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.91)