Research map / Reinforcement learning / Policy gradient and actor-critic
Ratio-Variance Regularized Policy Optimization
Yu Luo, Shuo Han, Yihan Hu, Lei Lv, Huaping Liu, Fuchun Sun, Jianye HAO, Dong Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace hard clipping (GRPO) with ratio-variance regularization (R2VPO-OFF) | GRPO · Math reasoning macro-average over 7 LLMs | accuracy: 42.50 -> 46.33 | Improved |
| use on-policy R2VPO instead of GRPO | GRPO · Qwen3-1.7B (fast) | accuracy: 30.70 -> 37.37 | Improved |
| use on-policy R2VPO instead of GRPO | GRPO · DeepSeek-Distill-Qwen-1.5B | accuracy: 29.23 -> 29.45 | No clear effect |
| add replay buffer off-policy reuse to R2VPO | R2VPO-ON · Math reasoning, 7 LLMs | accuracy: 44.90 -> 46.33 | Improved |
| add off-policy replay to R2VPO on openPangu-7B | R2VPO-ON · openPangu-7B | accuracy: 57.55 -> 56.32 | Hurt |
| increase replay buffer capacity (2, 4, 8) with GRPO (ablation) | GRPO · Qwen3-1.7B (fast) | test score | Hurt |
| increase replay buffer capacity (2, 4, 8) with R2VPO-OFF (ablation) | R2VPO-OFF · Qwen3-1.7B (fast) | test score | No clear effect |
| use adaptive dual update for lambda instead of fixed lambda=0.04 (ablation) | R2VPO-OFF · Qwen3-8B (fast) | test score | Improved |
| use fixed lambda=0.04 variance penalty instead of GRPO-CH clipping (ablation) | GRPO-CH · Qwen3-8B (fast) | test score | Improved |
| use R2VPO instead of PPO clipping | PPO · DeepMind Control Suite, 10 tasks | average score: 54.34 -> 65.11 | Improved |
| use R2VPO instead of PPO on sparse-reward Ball in Cup | PPO · DMC BallInCup | reward | Improved |
| use R2VPO instead of PPO on Cheetah Run | PPO · DMC CheetahRun | reward | Improved |
Closest papers by meaning
- Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.88)
- Simple Policy Optimization (ICML 2025 · similarity 0.88)
- Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.87)
- Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.87)
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.87)
- Embedding Safety into RL: A New Take on Trust Region Methods (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
