Research map / Large language models / Reasoning and chain of thought
Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that RL-trained reasoning models (DeepSeek-R1, QwQ-32B) show strong process-judgment ability on ProcessBench without PRM training, finds external PRMs fail to help them, and proposes Self-PRM (self-reranking via self-judgment) that gives modest best-of-N gains with low precision on hard problems.
Problem
Whether explicit process reward model (PRM) training is necessary for process-level reasoning judgment, given that pure RL on math problem-solving may already induce PRM-like capabilities; and whether existing PRMs help strong RL-trained reasoning models in reranking.
Approach
Argues that problem-solving and step-level judgment co-evolve under pure rule-based RL, evidenced by ProcessBench F1 of RL-trained models exceeding discriminative PRMs and by chi-square correlation tests; proposes Self-PRM, where a model uses its own judgments to rerank its sampled solutions in best-of-N.
Key results
- DeepSeek-R1 and QwQ-32B reach average ProcessBench F1 of 83.5 and 83.7, above discriminative PRMs such as Qwen2.5-Math-PRM-72B (78.3) and Skywork-PRM-7B (42.1).
- Qwen2.5-Math-PRM-72B reranking gives no gain over majority voting for QwQ-32B and DeepSeek-R1 on AIME24/AIME25/CNMO24.
- Self-PRM BoN lifts QwQ-32B on AIME24 from 86.7 (majority voting) to 90.0 at k=32, but on CNMO24 index-1 precision is only 2/55 (3.6%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with pure RL on math problem-solving without PRM supervision | RL-trained reasoning models · ProcessBench average | F1: DeepSeek-R1 83.5; QwQ-32B 83.7; Qwen2.5-Math-PRM-72B 78.3 | Helped |
| add Self-REF using self-generated solutions as references (ablation) | Qwen2.5-32B-Instruct · ProcessBench average | F1: 45.1 → 63.9 (+18.8) | Helped |
| add Self-REF using self-generated solutions as references (ablation) | R1-Distill-Qwen-32B · ProcessBench average | F1: 76.7 → 79.3 | Helped |
| add Self-REF using self-generated solutions as references (ablation) | QwQ-32B · ProcessBench average | F1: 83.7 → 83.0 | Hurt |
| add Self-REF using self-generated solutions as references (ablation) | DeepSeek-R1 · ProcessBench average | F1: 83.5 → 81.1 | Hurt |
| add Self-REF using self-generated solutions as references (ablation) | Qwen2.5-7B-Instruct · ProcessBench | F1: 43.9 → 58.7; 37.0 → 39.4; 29.5 → 25.7; 27.7 → 25.8 | Mixed |
| add Self-REF using self-generated solutions as references (ablation) | Qwen2.5-1.5B-Instruct · ProcessBench average | F1: 18.8 → 13.8 | Hurt |
| rerank sampled solutions with an external PRM instead of majority voting | QwQ-32B · AIME25 | accuracy: 76.7; 76.7 | No clear effect |
| rerank sampled solutions with Self-PRM instead of majority voting | QwQ-32B · AIME24 @32 | accuracy: 86.7 to 90.0 | Helped |
| use Self-PRM to select solutions on hard problems | QwQ-32B · CNMO24 index 1 | Self-PRM precision: 2 of the 55 selected solutions are truly correct; 3.6% | Hurt |
| use Self-PRM to select solutions on hard problems | DeepSeek-R1 · CNMO24 index 1 | Self-PRM precision: 10 of 39 selected solutions are true positives; 25.6% | Helped |
Surprising findings
- External Qwen2.5-Math-PRM-72B reranking is no better than majority voting for strong RL-trained reasoners.
- Self-REF (using own solutions as references) reduces ProcessBench F1 for QwQ-32B and DeepSeek-R1 while improving Qwen2.5-32B-Instruct by 18.8 F1.
Limitations
- Evidence is limited to math reasoning with correlational chi-square tests; the underlying mechanism is not investigated.
- Self-PRM shows low precision on hard problems, frequently accepting flawed solutions.
- Hard-problem precision analysis uses small problem subsets (e.g., 64 samples per problem) and few problems.
Experimental setup
Datasets: DAPO-Math-17k ProcessBench
Benchmarks: ProcessBench AIME24 AIME25 CNMO24
Metrics: F1 accuracy pass@k precision
Baselines: Qwen2.5-Math-PRM-72B Qwen2.5-Math-PRM-7B Skywork-PRM-7B Math-Shepherd-PRM-7B majority voting Pass@k
Builds on: DAPO ProcessBench DeepSeek-R1
Closest papers by meaning
- Unlocking Multimodal Mathematical Reasoning via Process Reward Model (NeurIPS 2025 · similarity 0.89)
- Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.89)
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.89)
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.89)
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.