Research map / Reinforcement learning / RL from feedback
Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes process advantage verifiers (PAVs) that predict step-level advantages under a complementary prover policy (Best-of-K of the base policy), and uses them as dense rewards for beam search and online RL on Gemma math reasoning.
Problem
Automated process reward models for LLM math reasoning have yielded only small gains (1-2%) over outcome reward models when used for test-time search or RL; it is unclear what per-step rewards should measure.
Approach
Define per-step process rewards as advantages under a separate prover policy (e.g., Best-of-K of the base policy) rather than Q-values or advantages of the base policy itself, combined with outcome reward as an effective reward; train verifiers (PAVs) to predict these advantages from Monte Carlo rollouts.
Key results
- Beam search with PAVs is >8% more accurate and 1.5-5x more compute efficient than best-of-N with an ORM.
- PAV-RL improves over ORM-RL by >7% test accuracy and is about 6x more sample efficient for Gemma 2B and 9B.
- A weaker 9B prover is more effective than a 27B prover for training PAVs for the 27B base policy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PAV effective rewards for beam search instead of ORM best-of-N | Gemma 2B, 9B, and 27B SFT policies · MATH | compute efficiency: 10× for Gemma-2B, 9B; 5× for Gemma-27B | Helped |
| Use PAV beam search instead of ORM best-of-N | Gemma 2B, 9B, and 27B SFT policies · MATH | accuracy: >8% | Helped |
| Add PAV process advantages as dense rewards to outcome-reward RL | Gemma 2B and Gemma 9B RFT policies · MATH | test accuracy: >7% | Helped |
| Add PAV process advantages as dense rewards to outcome-reward RL | Gemma 2B and Gemma 9B RFT policies · MATH | sample efficiency: 6× | Helped |
| Continue RFT training with PAV-RL | Gemma 2B and Gemma 9B · MATH | accuracy over RFT: 11% for 2B; 15% for 9B | Helped |
| Train with PAV-RL instead of ORM-RL | Gemma 2B · MATH | Pass @N: >7% | Helped |
| Use Qπ + αAμ instead of outcome reward alone | Planted sub-sequence task policy · Planted sub-sequence task | iterations to 100% accuracy: 350 iterations | Helped |
| Use Qπ + αAμ instead of outcome reward alone | Planted sub-sequence task policy · Planted sub-sequence task | Pass @N growth rate: 5× faster | Helped |
| Use PAV as the outcome re-ranker instead of a standard ORM (ablation) | Gemma 9B SFT policy · MATH, N=128 | accuracy: 4% | Helped |
| Add prover advantages Aμ to Qπ beam-search rewards (ablation) | Gemma 2B and Gemma 9B SFT policies · MATH | compute efficiency: 8× | Helped |
| Use Bo4 rather than Bo2 or Bo32 as the prover (ablation) | Gemma 2B SFT policy · MATH, beam search | beam-search performance | Helped |
| Use a weaker 9B prover rather than the 27B prover (ablation) | Gemma 27B SFT policy · MATH, N=16 | beam-search performance | Helped |
Surprising findings
- A weaker 9B prover outperforms the stronger 27B prover when used to train PAVs for the 27B base policy.
- Using the base policy's own advantages as process rewards in RL is equivalent to outcome-only RL policy gradients.
Limitations
- Verifier fitting errors upper-bound performance of the method.
- Prover policy is fixed during RL and automatic design of good provers is open.
- Theory assumes tabular softmax policies and oracle advantages.
Experimental setup
Datasets: MATH
Benchmarks: MATH
Metrics: accuracy compute efficiency pass@N
Baselines: outcome reward model best-of-N Q-value process reward model beam search ORM-RL RFT SFT
Builds on: REINFORCE rejection fine-tuning natural policy gradient process reward models Best-of-N re-ranking
Closest papers by meaning
- Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.94)
- Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.93)
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.92)
- ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs (NeurIPS 2025 · similarity 0.92)
- GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning (AAAI 2026 · similarity 0.91)
- Free Process Rewards without Process Labels (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.