Research map / Large language models / Reasoning and chain of thought

The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that arbitrary-order decoding in diffusion LLMs narrows reasoning potential (lower Pass@k) and proposes JustGRPO, which runs standard GRPO by treating the dLLM as an autoregressive policy during RL while keeping parallel decoding at inference.

Problem

Arbitrary-order generation in diffusion LLMs is assumed to expand reasoning potential, but RL for dLLMs on math and coding is burdened by intractable likelihoods over denoising trajectories; the paper asks whether arbitrary order actually helps reasoning under RL.

Approach

Diagnose "entropy degradation": confidence-based arbitrary-order decoding defers high-entropy logical connectives ("Therefore", "Since"), collapsing solution coverage. Then drop arbitrary order only during RL training, define an exact AR policy on top of the dLLM by masking future tokens, and apply standard GRPO without diffusion-specific adaptations; inference still uses parallel decoding.

Key results

• JustGRPO reaches 89.1% on GSM8K and 45.1% on MATH-500 at generation length 256, versus 78.6% and about 30% for LLaDA-Instruct base in the authors' reproduction.
• On HumanEval, AR-order Pass@k scales above arbitrary order; 21.3% of problems are solved only by AR versus 0.6% only by arbitrary order (Pass@1024).
• JustGRPO keeps parallel decoding, with the gap over LLaDA-Instruct on MBPP growing from +10.6 at 1 token/step to +25.5 at about 5 tokens/step.

Surprising findings

• Higher-entropy-preserving AR order yields better Pass@k than confidence-based arbitrary order even though arbitrary order matches or beats AR at k=1.
• Fixed random-permutation training (JustGRPO-Random) reaches only 82.2% on GSM8K, far below AR order.

Limitations

• Gains are reported on LLaDA-Instruct-based models only at 8B scale in the main results
• Comparisons with baselines rely partly on paper-reported numbers under heterogeneous settings
• Entropy degradation mechanism is shown empirically without a formal account

Experimental setup

Datasets: GSM8K MATH AceCoder-87K DiffuCoder selection (21K)

Benchmarks: GSM8K MATH-500 HumanEval MBPP MMLU MMLU-Pro HellaSwag ARC-C

Metrics: accuracy pass@k wall-clock time

Baselines: d1 LLaDOU LLaDA-1.5 wd1 d-TreeRPO ESPO GDPO SPG TraceRL

Builds on: GRPO LLaDA d1 ESPO DiffuCoder

Closest papers by meaning

• d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation (ICML 2026 · similarity 0.91)
• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• On the Reasoning Abilities of Masked Diffusion Language Models (ICLR 2026 · similarity 0.90)
• Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Models (ICML 2026 · similarity 0.89)
• Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning (ICML 2026 · similarity 0.89)
• UnMaskFork: Test-Time Scaling for Masked Diffusion via Deterministic Action Branching (ICML 2026 · similarity 0.89)