Research map / Reinforcement learning / RL from feedback
ExGRPO: Learning to Reason from Prior Successes
Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, Dongrui Liu, Jing Shao, Derek F. Wong, Yu Cheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add experience replay with ExGRPO to on-policy RLVR | On-policy GRPO · Qwen2.5-Math-7B reasoning avg | accuracy (%): 45.3 -> 48.3 | Improved |
| add ExGRPO on a weak backbone | On-policy RLVR · Llama3.1-8B | in-dist avg: 3.4 -> 6.1 | Improved |
| add ExGRPO on instruct model | On-policy RLVR · Qwen2.5-7B-Instruct | in-dist avg: 35.2 -> 39.3 | Improved |
| add ExGRPO on small model | On-policy RLVR · Qwen2.5-Math-1.5B | in-dist avg: 30.0 -> 35.3 | Improved |
| use continual ExGRPO instead of continual on-policy from LUFFY | LUFFY · in-dist avg | accuracy (%): 48.7 -> 51.4 | Improved |
| extend ExGRPO to continuous rewards | GRPO · WildChat-IF chat benchmarks | avg score: 20.0 -> 21.7 | Improved |
| use medium-difficulty Gaussian sampling (sigma=1) instead of narrower sigma=0.5 (ablation) | ExGRPO · OOD avg | accuracy (%): 54.8 -> 58.3 | Improved |
| use sigma=1 instead of wider sigma=1.5 (ablation) | ExGRPO · in-dist avg | accuracy (%): 46.2 -> 48.4 | Improved |
| use replay ratio 75% instead of 50% (ablation) | ExGRPO · training performance | accuracy | Hurt |
| add heuristic question selection instead of random selection (ablation) | ExGRPO · validation score | score | Improved |
| add policy shaping (ablation) | ExGRPO · validation score | score | Improved |
| apply shaping alone in on-policy RLVR (ablation) | On-policy RLVR · validation score | score | No clear effect |
Closest papers by meaning
- ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Learning to Reason under Off-Policy Guidance (NeurIPS 2025 · similarity 0.89)
- The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.89)
- Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.89)
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.89)
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
