Research map / Image generation / Fast diffusion sampling

Reconsidering Positional Supervision in Masked Diffusion Language Model Training

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that LLaDA-8B-Instruct is sensitive to tiny one-position token shifts during iterative decoding, and proposes CTC-S (CTC with a slack token and a blank-only collapse map) as an auxiliary SFT loss that improves open-ended generation over CE-only SFT.

Problem

Masked diffusion language models are trained with position-wise cross-entropy, which may make them sensitive to small positional shifts during iterative parallel decoding, a known issue for CE-trained non-autoregressive translation decoders.

Approach

Adapts CTC from NAT to MDLM SFT as an auxiliary loss, using a special slack token as the blank and a modified collapse map that removes only slack tokens so duplicate content tokens (e.g., "**", "100") are preserved.

Key results

• On LLaDA-8B-Instruct, CE + CTC-S scores 55.1 on Arena-Hard vs 50.0 for CE-Only (reference) and 43.7 for original LLaDA, with significant gains on all four open-ended benchmarks.
• Shifting 1% of generated tokens by one position drops Arena-Hard win rate against the unintervened model well below 50%; CTC-S stays at or above 50% under the same intervention.
• On LLaDA-MoE, CE + CTC-S reaches 63.7 on Arena-Hard versus 50.0 for CE-Only, and CTC-S beats a block-diffusion-swept CE-Only baseline on five of six comparisons.

Surprising findings

• Random slack-token augmentation alone (Jitter) gives mixed results, with Arena-Hard nearly overlapping CE-Only, so the alignment-flexible CTC-S objective drives most of the gain.
• CTC-S model emits slack tokens selectively, with only 51.9% of MT-Bench samples containing any slack and only 4.9% of emitted slack tokens at subword-internal positions.

Limitations

• Experiments limited to the SFT stage; effect of positional relaxation during pre-training is unexplored.
• Evaluated on LLaDA-family models trained from scratch; generalization to MDLMs like Dream with shift-operation training is untested.
• CTC-S loss weight lambda and max slack ratio were fixed without tuning.

Experimental setup

Datasets: Magpie-Pro-300K-Filtered Arena-Hard-v2.0 MT-Bench WildBench Creative Writing Bench v3 GPQA MMLU MBPP IFEval GSM8K

Benchmarks: Arena-Hard-v2.0 MT-Bench WildBench Creative Writing Bench v3 GPQA MMLU MBPP IFEval GSM8K

Metrics: win rate average score paired bootstrap 95% confidence interval

Baselines: LLaDA-8B-Instruct CE-Only Jitter block diffusion

Builds on: connectionist temporal classification LLaDA d1 codebase Magpie

Closest papers by meaning

• Mean-Field Parallel Decoding for Discrete Diffusion Language Models (NeurIPS 2026 · similarity 0.90)
• A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Curse (NeurIPS 2026 · similarity 0.89)
• StableMask: Refining Causal Masking in Decoder-only Transformer (ICML 2024 · similarity 0.88)
• UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models (ICLR 2026 · similarity 0.88)
• SimSD: Simple Speculative Decoding in Diffusion Language Models (NeurIPS 2026 · similarity 0.88)
• Latent Refinement Decoding: Enhancing Diffusion Language Models by Refining Belief States (NeurIPS 2026 · similarity 0.88)