Research map / Reinforcement learning / RL from feedback
Learning Unmasking Policies for Diffusion Language Models
Metod Jazbec, Theo X. Olausson, Louis Béthune, Pierre Ablin, Michael Kirchhof, Joao Monteiro, Victor Guilherme Turrisi da Costa, Jason Ramapuram, marco cuturi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a multiplicative correctness-times-speed reward instead of an additive penalty (ablation) | RL unmasking policy (alpha=1) · GSM8K training on LLaDA | training stability / reward hacking | Improved |
| train a policy for full diffusion (BL=256) instead of using confidence heuristics | LLaDA-8B-Instruct sampling · GSM8K, ~12 NFEs | accuracy (%): 30% -> 50% | Improved |
| add expert steering using Fast-dLLM semi-AR samples to full-diffusion RL training (ablation) | RL unmasking policy (BL=256) · GSM8K and MATH-500, mid-to-high NFEs | accuracy: approx. 80% GSM8K and 35% MATH | Improved |
| add expert steering to RL training (ablation) | RL unmasking policy · full diffusion training | training stability / controllability via alpha | Hurt |
| use RL policy instead of Fast-dLLM in semi-AR mid-to-high NFE range | LLaDA-8B-Instruct · GSM8K and MATH-500, BL=32 | accuracy vs NFEs | No clear effect |
| scale policy unmasking probabilities by beta at test time (ablation) | RL policy trained with alpha=1 · MATH-500 semi-AR, ~25 NFEs | accuracy: 10% -> 20% vs Fast-dLLM | Improved |
| use DPLS (dynamic Plackett-Luce) policy likelihood instead of Bernoulli (ablation) | RL unmasking policy · GSM8K, LLaDA | accuracy vs NFEs | No clear effect |
| feed top-50 confidences instead of max confidence per position (ablation) | RL unmasking policy · GSM8K, LLaDA | accuracy vs NFEs | Hurt |
| use dLLM hidden states as policy input (300M-parameter head) instead of confidences (ablation) | RL unmasking policy · GSM8K, LLaDA | accuracy and training stability | Hurt |
| zero out mask vector or time input at test time (ablation) | RL policy (alpha=1) · GSM8K and MATH-500, BL=32 | accuracy | Hurt |
| evaluate LLaDA-trained policies on Dream without retraining | RL unmasking policy · GSM8K, Dream-7B | accuracy vs NFEs | Mixed |
| transfer math-trained policies to coding without retraining | RL unmasking policy · HumanEval and MBPP | accuracy | Hurt |
Closest papers by meaning
- Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages (NeurIPS 2026 · similarity 0.91)
- Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking (NeurIPS 2025 · similarity 0.91)
- Re-evaluating Confidence Remasking in Masked Diffusion Language Models (NeurIPS 2026 · similarity 0.91)
- UnMaskFork: Test-Time Scaling for Masked Diffusion via Deterministic Action Branching (ICML 2026 · similarity 0.91)
- Learn from Your Mistakes: Self-Correcting Masked Diffusion Models (NeurIPS 2026 · similarity 0.90)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
