Research map / Large language models / Agents, code and math
Let's Verify Step by Step
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares outcome- and process-supervised reward models for math reasoning on MATH, finds process supervision is more reliable, uses active learning for step-level human labels, and releases the PRM800K dataset.
Problem
Training reliable reward models for multi-step mathematical reasoning: whether step-level (process) feedback is better than final-outcome feedback, given the high cost of human labels.
Approach
Collects 800K step-level human correctness labels on GPT-4-generated MATH solutions, selected by active learning toward convincing wrong-answer solutions, and trains a process reward model that scores solutions by the product of step correctness probabilities; compares it against outcome reward models at large and small scale, using a large PRM as a synthetic labeler for ablations.
Key results
- The process-supervised reward model solves 78.2% of a 500-problem MATH test subset under best-of-1860 search, outperforming the outcome-supervised RM and majority voting.
- Active learning gives an estimated 2.6x improvement in data efficiency for process supervision over uniform labeling.
- On 234 out-of-distribution STEM problems, the PRM reaches 72.9% aggregate vs 63.8% for the ORM and 61.3% for majority voting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use process supervision instead of outcome supervision | Reward models trained on small-scale generator samples · MATH, best-of-500 | accuracy | Helped |
| Use process supervision rather than outcome supervision | Large-scale reward model · MATH test subset, best-of-N | accuracy | Helped |
| Use process supervision rather than outcome supervision | Large-scale reward model · OOD STEM tests, best-of-100 | accuracy | Helped |
| Select convincing samples with active learning rather than uniform data labelling | Small-scale process-supervised reward model · MATH | data efficiency: 2.6x | Helped |
| Add RM-weighted voting to combine PRM scores and majority voting | PRM and majority-vote search · MATH test subset | accuracy | No clear effect |
| Iteratively retrain the selector between data-collection rounds (ablation) | Small-scale active-learning reward models · MATH | accuracy | No clear effect |
| Use product rather than minimum to reduce step scores, with neutral treated as positive (ablation) | PRM · MATH test subset, best-of-1860 | accuracy: 77.6% -> 78.2% | Helped |
| Treat neutral step labels as positive rather than negative, using product reduction (ablation) | PRM · MATH test subset, best-of-1860 | accuracy: 77.4% -> 78.2% | Helped |
| Treat neutral step labels as positive rather than negative, using minimum reduction (ablation) | PRM · MATH test subset, best-of-1860 | accuracy: 77.8% -> 77.6% | Hurt |
Surprising findings
- Using PRM_large for outcome supervision was more effective than final-answer checking in small-scale experiments, attributed to better handling of correct-answer solutions with incorrect reasoning.
- Iterative retraining of the selector PRM during active learning was unstable and gave no improvement over non-iterative selection.
Limitations
- Large-scale ORM and PRM are trained on non-comparable datasets, so the main comparison is confounded; small-scale ablations rely on a synthetic PRM_large labeler.
- Results are only shown in the math domain and may not generalize to other domains.
- Possible test set contamination from pretraining data, though decontamination was attempted.
Experimental setup
Datasets: MATH MathMix PRM800K
Benchmarks: MATH (500-problem test subset) AP Calculus AP Chemistry AP Physics AMC10/12
Metrics: best-of-N accuracy best-of-500 accuracy accuracy on held-out STEM tests
Baselines: outcome-supervised reward model majority voting final-answer outcome supervision RM-weighted voting
Builds on: token-level verifiers (Cobbe et al. 2021) Minerva Uesato et al. 2022 process supervision
Closest papers by meaning
- Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective (ICML 2025 · similarity 0.89)
- Free Process Rewards without Process Labels (ICML 2025 · similarity 0.88)
- Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.87)
- Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning (ICML 2024 · similarity 0.87)
- Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision (ICLR 2025 · similarity 0.87)
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.