Research map / Reinforcement learning / RL from feedback
Anchored Policy Optimization: Mitigating Exploration Collapse via Support-Constrained Rectification
Tianyi Wang, Long Li, Hongcan Guo, Yibiao Chen, Yixia Li, Yong Wang, Yun Chen, Guanhua Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GRPO with APO (anchored ratio rectification) | Qwen2.5-Math-7B · 5 math benchmarks avg | Pass@1: 48.68 -> 50.67 | Improved |
| replace GRPO with APO | Qwen2.5-Math-7B · 5 math benchmarks avg | Pass@K: 76.56 -> 78.07 | Improved |
| replace GRPO with APO | Llama-3.2-3B-Instruct · 5 math benchmarks avg | Pass@K: 43.82 -> 47.11 | Improved |
| replace GRPO with APO | Qwen2.5-7B · 5 math benchmarks avg | Pass@1: 42.68 -> 43.57 | Improved |
| add global KL regularization to GRPO (ablation) | GRPO on Qwen2.5-Math-7B · 5 math benchmarks avg | Pass@1: 48.68 -> 44.53 | Hurt |
| apply KL penalty only on errors instead of globally (ablation) | GRPO-KL on Qwen2.5-Math-7B · 5 math benchmarks avg | Pass@1: 44.53 -> 46.79 | Improved |
| error-only KL versus unregularized GRPO (ablation) | GRPO on Qwen2.5-Math-7B · 5 math benchmarks avg | Pass@1: 48.68 -> 46.79 | Hurt |
| add negative sample reinforcement (NSR) to GRPO | Qwen2.5-Math-7B · 5 math benchmarks avg | Pass@1: 48.68 -> 48.58 | No clear effect |
| remove pull anchor (beta = 0) (ablation) | APO on Qwen2.5-Math-7B · 5 math benchmarks avg | Pass@1: 50.67 -> 48.86 | Hurt |
| reduce anchor set size to K=4 from 8 (ablation) | APO on Qwen2.5-Math-7B · 5 math benchmarks avg | Pass@1: 50.67 -> 49.46 | Hurt |
| broaden anchor set to K=16 from 8 (ablation) | APO on Qwen2.5-Math-7B · 5 math benchmarks avg | Pass@1 | Hurt |
| push coefficient lambda 1.0 instead of 1.05 (ablation) | APO on Qwen2.5-Math-7B · 5 math benchmarks avg | Pass@1: 50.67 -> 49.24 | Hurt |
Closest papers by meaning
- Global Optimality for Constrained Exploration via Penalty Regularization (NeurIPS 2026 · similarity 0.88)
- Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.87)
- Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (NeurIPS 2026 · similarity 0.86)
- SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
- Occupancy-based Policy Gradient: Estimation, Convergence, and Optimality (NeurIPS 2024 · similarity 0.86)
- Parameter Exploration for RLVR via Variational Learning (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
