Research map / Large language models / Reasoning and chain of thought
Process reward models
Train process-level reward models with synthetic annotation and calibration to verify reasoning steps.
39 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2021–2026) · an approach in Reasoning and chain of thought, keyword cluster “test time · scaling · rl”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
1 papers in 2023–24 and 37 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 0.8% to 2.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2021 · Measuring Mathematical Problem Solving With the MATH Dataset (NeurIPS 2021 · 276 citations)
- 2024 · ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search (NeurIPS 2024 · 21 citations)
- 2025 · Generative Verifiers: Reward Modeling as Next-Token Prediction (ICLR 2025 · 2 citations)
- 2026 · GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning (AAAI 2026 · 2 citations)
Earliest papers here
- Measuring Mathematical Problem Solving With the MATH Dataset (NeurIPS 2021 · 276 citations)
- ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search (NeurIPS 2024 · 21 citations)
- Generative Verifiers: Reward Modeling as Next-Token Prediction (ICLR 2025 · 2 citations)
What has been tried
20 papers here report 224 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 9%, hurt 17%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add reference guidance when generating synthetic verification rationales | GSM8K · Best-of-N accuracy | 87.8% -> 91.7% | Helped | Generative Verifiers: Reward Modeling as Next-Token Prediction (ICLR 2025 · 2 citations) |
| balance positive and negative responses for each instruction | MATH-500 BoN · accuracy | 48.4; 49.0 | Mixed | Free Process Rewards without Process Labels (ICML 2025 · 1 citation) |
| Add AdaC curriculum regularization | six-benchmark average · accuracy | 68.3 -> 77.8 | Helped | AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasoners (NeurIPS 2025 · 1 citation) |
| Add majority voting@16 | RewardBench Overall · accuracy | 91.2 -> 91.9 | Helped | Reward Reasoning Models (NeurIPS 2025 · 1 citation) |
| Use finite differences instead of JVP to compute the antidistillation penalty | GSM8K · antidistillation performance | No clear effect | Antidistillation Sampling (NeurIPS 2025 · 1 citation) | |
| Increase trajectory-level reward weight alpha from 0.1 to 0.8 | AIME24 · accuracy | 26.7 -> 40.0 | Helped | ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs (NeurIPS 2025 · 1 citation) |
| Train ASPRM-M with mixed math and code data instead of math-only data | GSM8k · Bo64 | 86.35 | Helped | AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence (ICML 2025 · 0 citations) |
| use Q-RM instead of ORM with PPO | GSM8K training · sample efficiency | nearly 12× | Helped | Discriminative Policy Optimization for Token-Level Reward Models (ICML 2025 · 0 citations) |
| train on multi-domain CoTs instead of a same-size math-only subset | MMLU-Pro-CoT-Eval (Unlabeled), Math and non-Math · accuracy | Helped | VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data (ICML 2025 · 0 citations) | |
| Scalar process rewards added directly to GRPO (Variants 1 and 2) | training dynamics / test accuracy · test accuracy | Hurt | Unlocking Multimodal Mathematical Reasoning via Process Reward Model (NeurIPS 2025 · 0 citations) | |
| Have Command A annotate markers on-the-fly | AlpacaEval-LI · Violation (%) | 24.74% → 0.75% | Helped | Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers (NeurIPS 2025 · 0 citations) |
| Add TTPL-designed projection instead of using KNN without projection | TSP1K–TSP100K · optimality gap | 4.49% -> 2.65%; 12.80% -> 3.25%; 17.42% -> 3.63%; 48.00% -> 3.29%; 121.42% -> 3.22% | Helped | Improving Generalization of Neural Combinatorial Optimization for Vehicle Routing Problems via Test-Time Projection Learning (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 37 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MATH · 12 of 37 papers
- GSM8K · 12 of 37 papers
- PRM800K · 6 of 37 papers
- MATH500 · 4 of 37 papers
- ProcessBench · 4 of 37 papers
- LiveCodeBench · 3 of 37 papers
- MMLU · 3 of 37 papers
- Math-Shepherd · 3 of 37 papers
Baselines
- Qwen2.5-Math-PRM-7B · 7 of 37 papers
- Math-Shepherd · 6 of 37 papers
- Math-Shepherd-PRM-7B · 5 of 37 papers
- Majority Vote · 5 of 37 papers
- ORM · 4 of 37 papers
- Qwen2.5-Math-PRM-72B · 4 of 37 papers
- Majority Voting · 3 of 37 papers
- LLM-as-a-Judge · 3 of 37 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 38 papers; from OpenAlex and the papers' affiliations).
- Microsoft · 5 papers
- Tsinghua University · 4 papers
- ByteDance · 3 papers
- Tencent · 3 papers
- University of Wisconsin - Madison · 2 papers
- Korea Advanced Institute of Science and Technology · 2 papers
- Peking University · 2 papers
Use it
Download BibTeX of 21 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Efficient and adaptive thinking (231 papers)
- Test-time compute scaling (205 papers)
- RL with verifiable rewards for reasoning (270 papers)
- Routing, pruning and distillation for reasoning (135 papers)
Elsewhere in the topic
- Long CoT dynamics and efficiency (136 papers)
- Chain-of-thought training and analysis (127 papers)
- Efficient LLM serving (126 papers)
- Search and self-correction in reasoning (123 papers)
- Speculative decoding (92 papers)
- KV cache compression (45 papers)
- Theory of autoregressive reasoning (28 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).