Research map / Large language models / Reasoning and chain of thought
Lookahead Unmasking Elicits Reliable Decoding in Diffusion Language Models
Sanghyun Lee, Seungryong Kim, Jongho Park, Dongmin Park
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Lookahead Unmasking (LookUM), an inference-time method that chooses which tokens to unmask in masked diffusion language models by looking ahead rather than greedily using local confidence.
Problem
Masked diffusion language models depend heavily on the order in which tokens are unmasked at inference; confidence-based greedy ordering is short-sighted and lets early decoding errors cascade.
Approach
Replace local confidence-based unmasking order with a lookahead procedure that accounts for downstream consequences of unmasking choices, so that early decoding errors are avoided and extra test-time computation is used.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LookUM lookahead path selection | LLaDA-8B · HumanEval (len 128) | accuracy (%): 19.5 -> 27.4 | Improved |
| add LookUM lookahead path selection | LLaDA-8B · GSM8K (len 128) | accuracy (%): 68.3 -> 72.7 | Improved |
| add LookUM to RL-tuned model | LLaDA-1.5 · GSM8K (len 256) | accuracy (%): 79.4 -> 82.3 | Improved |
| add LookUM | LLaDA-1.5 · Countdown (len 256) | accuracy (%): 23.4 -> 17.9 | Hurt |
| add remasking (ReMDM) to confidence unmasking | LLaDA-8B · HumanEval (len 128) | accuracy (%): 19.5 -> 17.1 | No clear effect |
| use average negative entropy instead of average confidence verifier (ablation) | LookUM · GSM8K (len 128) | accuracy (%): 71.0 -> 72.3 | Improved |
| use certainty filtering pool instead of N-best (ablation) | LookUM · Countdown (len 128) | accuracy (%): 32.0 -> 19.5 | Hurt |
| use NIS instead of SMC sampling (ablation) | LookUM · Countdown (len 128) | accuracy (%): 23.1 -> 31.3 | Improved |
| use NIS instead of SMC sampling (ablation) | LookUM · MATH500 (len 128) | accuracy (%): 29.0 (SMC) vs 28.6 (NIS) | No clear effect |
| use external process reward model verifier instead of uncertainty (ablation) | LookUM · MATH500 / GSM8K | accuracy (%): 26.0 -> 24.0 (MATH500 SMC 16 particles) | Hurt |
| increase number of paths beyond 2-4 (ablation) | LookUM · GSM8K/MATH500 | accuracy (%) | No clear effect |
Experimental setup
Baselines: confidence-based sampling
Builds on: masked diffusion models
Closest papers by meaning
- Lookahead Path Likelihood Optimization for Diffusion LLMs (ICML 2026 · similarity 0.92)
- UnMaskFork: Test-Time Scaling for Masked Diffusion via Deterministic Action Branching (ICML 2026 · similarity 0.92)
- Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking (NeurIPS 2025 · similarity 0.91)
- Re-evaluating Confidence Remasking in Masked Diffusion Language Models (NeurIPS 2026 · similarity 0.91)
- CORE: Context-Robust Remasking for Diffusion Language Models (ICML 2026 · similarity 0.91)
- Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
