Research map / Image generation / Fast diffusion sampling

A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Curse

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Theoretically explains why masked diffusion LMs mitigate the reversal curse via position-invariant value-side storage of token-pair evidence plus relative-position attention routing, with one-layer proofs and RADD and LLaDA/Dream experiments.

Problem

Why masked diffusion language models (MDMs) show a much weaker reversal curse than autoregressive models, beyond the any-order training objective explanation.

Approach

Shows forward masked training increases a position-invariant storage term Y_{A,B} shared across forward and reverse configurations, while relative positional encodings (RoPE, ALiBi) keep forward and reverse attention routes positively correlated, so forward gradients align with reverse loss gradients to first order.

Key results

• In the one-layer setting, MDM reverse accuracy reaches 33.9-55.7% across L=10-40 while the ARM reverse accuracy is 0.00%.
• Full-parameter forward-reverse gradient cosine stays above 0.6 during one-layer training.
• Under RoPE, the expected attention-score correlation lower bound is (1/D)Tr(R(2Δ)), with minimum 0.337 for D=64 over Δ≤50.

Surprising findings

• Learned absolute embeddings fit the forward direction but transfer poorly to reverse queries (11.5% reverse at L=10), while ALiBi transfers almost perfectly (99.8%).

Limitations

• Analysis is for a simplified one-layer MDM; multi-layer results are empirical only.
• Reverse probabilities remain below forward probabilities in large-scale experiments.
• Full-gradient alignment is not proven pointwise; only the Y-column component has a sign-determined result.

Experimental setup

Datasets: Parent-Child dataset synthetic one-layer lowercase-uppercase pair sequences

Benchmarks: Parent-Child reversal evaluation

Metrics: accuracy attention score correlation gradient cosine target probability

Baselines: GPT-2-style ARM learned absolute embeddings RoPE ALiBi

Builds on: RADD RoPE ALiBi Reversal curse analysis (Zhu et al.)

Closest papers by meaning

• Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.91)
• Re-evaluating Confidence Remasking in Masked Diffusion Language Models (NeurIPS 2026 · similarity 0.90)
• Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions (ICML 2025 · similarity 0.90)
• CORE: Context-Robust Remasking for Diffusion Language Models (ICML 2026 · similarity 0.90)
• Support Before Frequency in Discrete Diffusion (NeurIPS 2026 · similarity 0.90)
• Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models (NeurIPS 2026 · similarity 0.89)