Research map / Reinforcement learning / RL from feedback
Accelerating RL for LLM Reasoning with Optimal Advantage Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes A*-PO, a two-stage RL method that estimates the optimal value V* offline from reference-policy samples and then performs single-sample-per-prompt least-squares advantage regression, with theoretical guarantees and reduced training time and memory on math reasoning.
Problem
RL fine-tuning of LLMs for reasoning (PPO, GRPO) is computationally and memory expensive due to critic networks or multiple generations per prompt, and relies on current-policy advantage estimates.
Approach
Estimate the KL-regularized optimal value V* offline by averaging exp(reward/beta) over N reference-policy samples per prompt, then regress the log-ratio beta*ln(pi/pi_ref) onto the optimal advantage r - V_hat using only one on-policy sample per prompt, with no clipping or reward normalization.
Key results
- A*-PO reduces training time to 6.92 hours for Qwen2.5-1.5B versus 15.01 for GRPO and 10.84 for PPO on MATH, and achieves the lowest KL (0.069).
- On MATH500 for Qwen2.5-7B, A*-PO reaches 76.2 versus 74.4 (PPO) and 73.2 (GRPO), with training time 11.01 hours versus 20.15 (GRPO).
- On long-context OpenR1 with DeepSeek-distilled Qwen-1.5B, A*-PO averages 21.57 Avg@32 across AIME24/25 and HMMT, slightly above PPO (21.52) and GRPO (20.97).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use A★-PO instead of PPO, GRPO, or REBEL | Qwen2.5 models on MATH benchmarks · MATH500, Minerva Math, OlympiadBench, AMC 23 Avg@32 | accuracy | Mixed |
| Use A★-PO instead of PPO or GRPO | DeepSeek-R1-Distill-Qwen-1.5B, long-context training · AIME 2024/2025 and HMMT February 2024/2025 | Avg@32: 21.52 -> 21.57 (average) | Mixed |
| Increase stage-1 generation count N (ablation) | A★-PO, Qwen2.5-3B · MATH | MATH500 accuracy | Helped |
| Filter prompts whose N sampled responses are all incorrect (ablation) | A★-PO · MATH500 | accuracy | No clear effect |
| Filter prompts whose N sampled responses are all incorrect, with N=8 (ablation) | A★-PO · MATH500 | training time: 28% | Helped |
| Decrease β₁ for estimating V★, down to β₁=1/8 (ablation) | A★-PO · MATH500 | accuracy | Helped |
| Decrease β₁ further below 1/8 (ablation) | A★-PO · MATH | performance | Hurt |
| Add format-based reward rules (ablation) | rule-based reward training · Experiments | performance | No clear effect |
Limitations
- REBEL runs out of memory in the long-context setting, so it is excluded there.
- Long-context experiments use only a 5k-problem subset of OpenR1 due to compute limits.
- Theoretical guarantees rely on realizability and a Pass@k lower bound assumption (E[r] >= alpha > 0 under the reference policy).
Experimental setup
Datasets: GSM8K MATH OpenR1-Math-220K
Benchmarks: GSM8K MATH500 Minerva Math OlympiadBench AMC 23 AIME 24 AIME 25 HMMT Feb 24 HMMT Feb 25
Metrics: accuracy Avg@32 Pass@32 training time peak memory KL divergence
Baselines: PPO GRPO REBEL
Builds on: PPO GRPO REBEL DeepSeek-R1 recipe
Closest papers by meaning
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.92)
- Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (ICLR 2026 · similarity 0.92)
- Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.92)
- Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.92)
- Tricks or Traps? A Deep Dive into RL for LLM Reasoning (ICLR 2026 · similarity 0.92)
- Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.