Research map / Reinforcement learning / Policy gradient and actor-critic
DisPPO: Quantile-Based Distributional Reinforcement Learning for Large Language Models
Zhijian Zhou, Long Li, Xuan Zhang, Zongkai Liu, Yanting Miao, Yuchen Liu, Deshu Chen, Ke Li, Xing Sun, Ruoxi Jiang, Xiaoyu Tan, Chao Qu and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace scalar critic with quantile distributional critic | PPO · Math reasoning, average | avg@k: 43.1 -> 45.0 | Improved |
| replace scalar critic with quantile distributional critic | PPO · Math reasoning, average | pass@k: 67.3 -> 71.2 | Improved |
| replace scalar critic with quantile distributional critic | PPO · AIME24 | pass@256: 73.3 -> 86.7 | Improved |
| replace scalar critic with quantile distributional critic | PPO · College Math | avg@8: 42.7 -> 43.5 | Improved |
| replace scalar critic with quantile distributional critic | PPO · Bird (in domain) | greedy accuracy (%): 61.4 -> 63.5 | Improved |
| replace scalar critic with quantile distributional critic | PPO · Spider (out of domain) | greedy accuracy (%): 77.0 -> 81.2 | Improved |
| replace scalar critic with quantile distributional critic | PPO · Multi-turn tool reasoning, average | avg@64: 29.1 -> 30.2 | Improved |
| replace scalar critic with quantile distributional critic | PPO · RLHF UltraFeedback: AlpacaEval 2 | win rate (%): 20.77 -> 30.14 | Improved |
| replace scalar critic with quantile distributional critic | PPO · RLHF UltraFeedback: Arena-Hard | score: 18.3 -> 19.3 | Improved |
| replace scalar critic with quantile distributional critic | PPO · RLHF UltraFeedback: MT-Bench | score: 7.18 -> 7.41 | Improved |
| replace scalar critic with quantile distributional critic | PPO · AIME25 value prediction | MAE (lower better): 0.3016 -> 0.2749 | Improved |
| replace scalar critic with quantile distributional critic, middle MC bin [0.3, 0.7) | PPO · AIME25 value prediction by MC bin | MAE (lower better): 0.1723 -> 0.2346 | Hurt |
Closest papers by meaning
- DAPO : Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization (NeurIPS 2025 · similarity 0.90)
- Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions (NeurIPS 2025 · similarity 0.89)
- Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.89)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.88)
- SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning (ICML 2026 · similarity 0.88)
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
