Research map / Image generation / Fast diffusion sampling

Multi-Token Residual Prediction

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Multi-token Residual Prediction (MRP), a lightweight 3-layer module that predicts inter-step logit residuals of block diffusion LMs (SDAR) to enable speculative/direct decoding for speedup and a remasking scheme that recovers accuracy under low-threshold dynamic decoding.

Problem

Diffusion language models face a quality-throughput tradeoff: unmasking more tokens per denoising step increases speed but degrades quality because tokens are decoded without awareness of each other.

Approach

Predict the small logit residual between adjacent denoising steps from backbone hidden states with a lightweight module, avoiding a second backbone forward; use it as a drafter for lossless speculative decoding in static regime and as a remasking signal in dynamic threshold regime.

Key results

• Speculative MRP (K=3) on SDAR-8B GSM8K gives 1.40x throughput with accuracy preserved (90.4 vs 90.9 base).
• MRP Remask at tau=0.5 improves SDAR-8B HumanEval from 32.3 to 54.9 (+22.6) and SDAR-4B GSM8K from 63.4 to 81.1 (+17.7).
• Residual modeling outperforms direct distillation at K=4 by 12.9 to 73.1 GSM8K points across scales.

Surprising findings

• Direct distillation of full log-density collapses at K>=2 on 4B and 8B, while residual targets remain stable across MRP steps.
• Speculative decoding's position-wise verification keeps quality lossless, while remasking with the same threshold yields small regressions on code at conservative thresholds.

Limitations

• Larger quality degradation on code generation than reasoning, attributed to sharper distributional shifts.
• Fixed number of MRP steps K throughout generation; no adaptive schedule.
• Remasking adds an MRP forward per step, reducing throughput relative to dynamic decoding.

Experimental setup

Datasets: SFTDatasetV3 OpenHermes-2.5 GenQA Infinity-Instruct GSM8K MATH500 HumanEval MBPP

Benchmarks: GSM8K MATH500 HumanEval MBPP

Metrics: accuracy tokens per second speedup

Baselines: Fast-dLLM threshold decoding WINO SSD Direct modeling distillation Base r=1 static decoding

Builds on: SDAR EAGLE Multi-Token Prediction Self Speculative Decoding WINO Fast-dLLM

Closest papers by meaning

• Mean-Field Parallel Decoding for Discrete Diffusion Language Models (NeurIPS 2026 · similarity 0.89)
• Denoising Time Matters: Diverse Generation in Diffusion Language Models (NeurIPS 2026 · similarity 0.89)
• MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization (NeurIPS 2025 · similarity 0.88)
• Better & Faster Large Language Models via Multi-token Prediction (ICML 2024 · similarity 0.88)
• Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing (ICML 2026 · similarity 0.88)
• Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion (NeurIPS 2026 · similarity 0.88)