Research map / Image generation / Fast diffusion sampling

Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ADAS, a training-free greedy reranking rule that discounts each masked position's confidence by its attention to already-selected uncertain positions, plugged into Top-k, Fast-dLLM and EB-Sampler for masked diffusion LMs.

Problem

Parallel unmasking in masked diffusion language models is fragile because token-wise confidence ranking ignores dependencies among positions committed in the same step, hurting quality in the low-NFE regime.

Approach

Keep each base sampler's stopping rule but replace confidence ranking with a greedy attention-discounted utility: c_i minus alpha times the sum over selected s of A_is(1-c_s), so candidates strongly attending to uncertain selected positions are deferred rather than hard-excluded.

Key results

• Averaged over datasets, ADAS gives matched-NFE gains of +9.11 points on LLaDA-8B-Base and +10.46 on Dream-7B-Base in the low-NFE regime across the three samplers.
• Across 90 matched operating points, mean gain is +9.27 (bootstrap 95% interval [+7.74, +10.84]), improving in 80 and regressing in 10.
• Runtime overhead from attention extraction and reranking is 3.1% per model forward on average.

Surprising findings

• Attention-discounted reranking delays the confidence-preferred token in favor of replacements with lower KL change, which the authors interpret as stability-driven selection.
• Gains are smallest on MATH500 (matched gain +5.73 in robustness analysis), which the authors attribute to sequential chains rather than pairwise incompatibility.

Limitations

• Attention is an imperfect proxy for dependence; it may reflect syntax, position, or formatting and omits higher-order interactions.
• Evaluated only on two 7-8B base MDLMs and reasoning/code tasks; alpha=40 chosen on one model/benchmark setting.
• Commitments are irreversible; no revise-or-remask mechanism.

Experimental setup

Datasets: GSM8K MATH500 HumanEval MBPP

Benchmarks: GSM8K MATH500 HumanEval MBPP

Metrics: accuracy pass@1 NFE matched-NFE gain

Baselines: Top-k Fast-dLLM EB-Sampler KLASS DAPD

Builds on: Fast-dLLM EB-Sampler KLASS

Closest papers by meaning

• Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking (NeurIPS 2025 · similarity 0.92)
• Re-evaluating Confidence Remasking in Masked Diffusion Language Models (NeurIPS 2026 · similarity 0.91)
• Encoder-Decoder Diffusion Language Models for Efficient Training and Inference (NeurIPS 2025 · similarity 0.91)
• CORE: Context-Robust Remasking for Diffusion Language Models (ICML 2026 · similarity 0.90)
• Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.90)
• Remasking Discrete Diffusion Models with Inference-Time Scaling (NeurIPS 2025 · similarity 0.90)