Research map / Image generation / Fast diffusion sampling

Attention-Based Sampler for Diffusion Language Models

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Attn-Sampler, a training-free dLLM sampling-order method that decodes masked tokens in descending attention column-sum order, with dynamic attention thresholding for parallel decoding.

Problem

Choosing the token sampling order in diffusion LLMs to maximize target sequence log-likelihood; existing samplers (confidence, margin, entropy, thresholding, top-k) are token-level greedy heuristics without theoretical grounding.

Approach

Frames order selection as minimizing a permutation dependency gap, shows NP-hardness via weighted linear ordering, replaces it with a rank-based expected approximation, and proves the tractable bound is minimized by decoding in descending total attention (attention column sum) order; adds a dynamic attention threshold for parallel decoding.

Key results

• Attn-Sampler (Sequential) reaches 57.67% average on Fast-dLLM v2 7B, versus 54.66% for the top confidence sampler and 56.57% for the entropy sampler.
• On LLaDA-1.5 8B, Attn-Sampler (Sequential) averages 52.84% versus 50.98% for confidence sampling.
• On GSM8K with Fast-dLLM v2 7B, Attn-Sampler achieves 84.2% accuracy at 95 TPS, while Fast-dLLM gets 82.1% at the same throughput.

Surprising findings

• Static or top-k attention-based parallelization degrades accuracy sharply at higher parallelism (top-k accuracy falls from 81.12 to 65.35 as throughput rises from 67 to 124 TPS).

Limitations

• Cannot be directly extended to uniform diffusion, as the optimal permutation relies on one-pass decoding of masked dLLMs.
• Theoretical bound relies on first-order approximations and assumptions that rarely hold exactly in practice, as the authors acknowledge.
• Evaluated only on four benchmarks and three dLLMs; no error bars reported.

Experimental setup

Datasets: GSM8K MATH HumanEval MBPP

Benchmarks: GSM8K MATH HumanEval MBPP

Metrics: accuracy tokens per second (TPS)

Baselines: top-1 confidence sampler top-1 margin sampler top-1 entropy sampler Fast-dLLM EB-Sampler KLASS

Builds on: Fast-dLLM KLASS EB-Sampler LLaDA

Closest papers by meaning

• Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.91)
• Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models (NeurIPS 2026 · similarity 0.89)
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.89)
• Free Lunch for Pass@$k$? Low Cost Diverse Sampling for Diffusion Language Models (NeurIPS 2026 · similarity 0.89)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.88)
• Theoretical Benefit and Limitation of Diffusion Language Model (NeurIPS 2025 · similarity 0.88)