Research map / Large language models / Reasoning and chain of thought
Free Process Rewards without Process Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that training an outcome reward model with log-likelihood-ratio parameterization yields an implicit process reward model for free, and that this PRM, trained on response-level labels only, beats MCTS-based Math-Shepherd on MATH best-of-N at a fraction of the cost.
Problem
Training process reward models (PRMs) requires step-level labels, which are expensive to collect via MCTS-style rollouts or verifiers, limiting PRM scaling.
Approach
Parameterize the outcome reward as beta times the log-likelihood ratio of policy to reference model; then the cumulative sum of per-token log-ratios over a prefix equals a Q value, so step rewards (differences of Q) come for free when training an ORM on response-level labels with any objective (DPO, KTO, NCA, CE).
Key results
- Implicit PRM (DPO) reaches an averaged best-of-N accuracy of 50.4 across three generators, above Math-Shepherd reimplementation (47.8) and off-the-shelf RLHFlow PRMs (49.1).
- Implicit PRM (CE) averages 48.4 and is about 38.8x cheaper in FLOPs than Math-Shepherd, using less than 1/38 of the training data.
- Training on Math-Shepherd step labels added to the implicit PRM gives no consistent gain (avg 49.2 vs 49.3 without).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train implicit PRMs using response-level labels with DPO, KTO, NCA, or CE | Llama-3.1-8B-Instruct implicit PRM · MATH-500 best-of-N across three generation models | accuracy: DPO 50.4; NCA 49.4; CE 48.4 | Helped |
| use CE rather than the implemented Math-Shepherd baseline | MATH-500 BoN · average across three generation models and @4/@16/@64 | accuracy: 48.4; 0.6 | Helped |
| balance positive and negative responses for each instruction (ablation) | CE implicit PRM · MATH-500 BoN | accuracy: 48.4; 49.0 | Mixed |
| balance classes across the entire dataset by filtering examples (ablation) | CE implicit PRM · MATH-500 BoN | accuracy: 48.4; 47.8 | Hurt |
| add task-irrelevant UltraFeedback instructions (ablation) | DPO implicit PRM · MATH-500 BoN | average accuracy: 49.3; 49.2 | Hurt |
| add coding instructions from UltraInteract (ablation) | DPO implicit PRM · MATH-500 BoN | average accuracy: 49.3; 49.2 | Hurt |
| deduplicate training responses (ablation) | DPO implicit PRM · MATH-500 BoN | average accuracy: 49.3; 47.6 | Hurt |
| replace part of original rollouts with base-model-generated rollouts (ablation) | DPO implicit PRM · MATH-500 BoN | average accuracy: 49.3; 48.7 | No clear effect |
| add a second training stage using Math-Shepherd step labels (ablation) | DPO implicit PRM · MATH-500 BoN | average accuracy: 49.3; 49.2 | No clear effect |
| remove the reference model at inference (ablation) | DPO implicit PRM trained with a reference model · MATH-500 BoN | average accuracy: 50.4; 50.6 | No clear effect |
| remove the reference model during training and inference (ablation) | DPO implicit PRM · MATH-500 BoN | average accuracy: 50.4; 50.3 | No clear effect |
| increase the number of training responses per instruction (ablation) | Implicit PRM · MATH-500 BoN | accuracy | Helped |
Surprising findings
- Adding step-level labels from Math-Shepherd to the implicit PRM does not improve best-of-N performance.
- Removing the reference model at training and inference does not hurt best-of-N accuracy, and the reference model Llama-3.1-8B-Instruct alone is already a competitive reward model.
- Stronger best-of-N reranking ability does not translate into better policy performance; DPO and CE implicit PRMs lower MATH500 accuracy as policies.
Limitations
- Relies on a reference model at inference, roughly doubling reward-model compute, although the total cost is marginal when the generator is large.
- Evaluated mainly on MATH with Llama-3.1 and Mistral models; generalization to other domains is untested.
- Gains from step labels are inconclusive since the MCTS-based annotations are themselves noisy.
Experimental setup
Datasets: UltraInteract UltraFeedback MATH
Benchmarks: MATH-500
Metrics: best-of-N accuracy (@4, @16, @64) average accuracy FLOPs GPU time cost
Baselines: Math-Shepherd AutoPSV EurusRM-7B SkyworkRM-Llama3.1-8B ArmoRM-Llama3-8B RLHFlow-8B-Mistral-Data RLHFlow-8B-DS-Data
Builds on: DPO Math-Shepherd AutoPSV KTO NCA
Closest papers by meaning
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.91)
- Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.91)
- Process Reward Model with Q-value Rankings (ICLR 2025 · similarity 0.90)
- Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning (AAAI 2026 · similarity 0.89)
- AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence (ICML 2025 · similarity 0.89)
- Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.