Research map / Reinforcement learning / RL from feedback
Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
Jingyang Ou, Jiaqi Han, Minkai Xu, Shaoxuan Xu, Jianwen Xie, Stefano Ermon, Yi Wu, Chongxuan Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| sequence-level ELBO policy optimization (ESPO) instead of token-level d1/wd1 | LLaDA-8B-Instruct · Countdown | accuracy: 48.3 -> 81.0 (wd1 vs ESPO avg) | Improved |
| ESPO instead of token-level baselines | LLaDA-8B-Instruct · Sudoku | accuracy: 23.1 -> 86.0 (wd1 vs ESPO avg) | Improved |
| ESPO instead of d1/wd1 | LLaDA-8B-Instruct · GSM8K | accuracy: 80.1 -> 82.0 (wd1 vs ESPO avg) | Improved |
| ESPO instead of d1/wd1 | LLaDA-8B-Instruct · MATH | accuracy: 37.7 -> 39.5 (d1 vs ESPO avg) | Improved |
| ESPO instead of d1/wd1 | Dream-7B-Instruct · GSM8K / MATH | accuracy: 80.5 -> 81.3 GSM8K; 45.8 -> 46.0 MATH (wd1 vs ESPO) | Improved |
| ESPO instead of d1/wd1 | LLaDA-8B-Instruct · HumanEval / MBPP | pass@1: HumanEval 37.8 -> 40.1; MBPP 37.8 -> 45.4 (vs base LLaDA) | Improved |
| ESPO on Dream Sudoku in 24-token direct-answer setting | Dream-7B-Instruct · Sudoku (Dream setting) | accuracy: 98.1 -> 98.0 | No clear effect |
| sequence-level action space with ELBO (vs token-level ELBO or mean-field) (ablation) | ESPO · Sudoku training reward | reward | Improved |
| use k2 KL estimator instead of k1 or k3 (ablation) | ESPO · Sudoku training reward | reward | Improved |
| increase Monte Carlo samples from 1 to 2 or 4 (ablation) | ESPO · Sudoku training reward | reward | Improved |
| increase Monte Carlo samples (ablation) | ESPO · Countdown training reward | reward | No clear effect |
| vary policy update value mu (8, 12, larger) (ablation) | ESPO · Countdown and Sudoku | reward | No clear effect |
Closest papers by meaning
- Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages (NeurIPS 2026 · similarity 0.93)
- d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- ESRL: Efficient Sampling-Based Reinforcement Learning for Sequence Generation (AAAI 2024 · similarity 0.89)
- Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening (ICML 2026 · similarity 0.89)
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.89)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
