Research map / Reinforcement learning / Policy gradient and actor-critic
Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
Youssef Mroueh, Nicolas Dupuis, Brian Belgodere, Apoorva Nitsure, Mattia Rigotti, Kristjan Greenewald, Jiri Navratil, Jarret Ross, Jesus Rios
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| mask zero-variance samples in on-policy GRPO (ablation) | on-policy GRPO · GSM8K test | Pass@1 (%): 45 -> 50 | Improved |
| use off-policy GRPO (server update every v=10, i=1) instead of on-policy | GRPO · GSM8K test | Pass@1 (%): 45 -> 50 | Improved |
| reuse fixed batch for 10 SGD iterations (v1,i10) off-policy variant (ablation) | GRPO · GSM8K test | convergence speed | Hurt |
| use off-policy GRPO (v10,i1) instead of on-policy | GRPO · Aime24 | Pass@1 mean: 0.3022 -> 0.3049 | No clear effect |
| use off-policy GRPO (v10,i1) instead of on-policy | GRPO · Math500 | extractive match mean: 0.8519 -> 0.8474 | No clear effect |
| reset reference policy to latest model each stage (ablation) | GRPO · GSM8K test | Pass@1 | Improved |
Closest papers by meaning
- Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends (ICLR 2026 · similarity 0.93)
- Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation (ICML 2026 · similarity 0.92)
- TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback (NeurIPS 2026 · similarity 0.90)
- $f$-GRPO & Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment (NeurIPS 2026 · similarity 0.89)
- Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO (NeurIPS 2026 · similarity 0.89)
- Optimistic Model Rollouts for Pessimistic Offline Policy Optimization (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
