Research map / Reinforcement learning / RL from feedback
Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PURE, a min-form credit assignment for process-reward-model-based RL fine-tuning of LLMs that replaces cumulative summed future rewards with the minimum future step reward, avoiding reward hacking and training collapse.
Problem
Process reward models (PRMs) are effective for test-time scaling but cause reward hacking and training collapse when used for reinforcement fine-tuning of LLMs on reasoning tasks, largely due to summation-form credit assignment.
Approach
Replace the canonical summation-form value (cumulative gamma-decayed future rewards) with a min-form credit assignment that values a response by its worst step, implemented by a softmax-weighted transform of process rewards (temperature T) so that the rest of the PPO/RLOO pipeline is unchanged; combined with a small fraction (10%) of verifiable rewards.
Key results
- PURE-PRM+VR on Qwen2.5-Math-7B reaches 53.3 average over 5 math benchmarks and 82.5 on AMC23, the best among compared models in the paper.
- PURE-PRM (min-form) reaches 49.3 average on Qwen2.5-Math-7B vs 48.3 for PURE-VR, while sum-form PRM training collapses around step 25.
- PURE-PRM-7B achieves 57.5 average F1 on ProcessBench, above the previous best 56.5 reported in that benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace summation-form with min-form credit assignment for PRM rewards (ablation) | PURE-PRM and PURE-PRM+VR · Qwen2.5-Math series, training | training stability: sum-form ... collapse at step 25; min-form methods remain stable over 200 steps | Helped |
| Replace summation-form with min-form credit assignment (ablation) | PURE-PRM · 5-benchmark average | pass@1 accuracy: around 30 -> 49.3 | Helped |
| Add verifiable rewards to min-form process-reward training | PURE-PRM · Qwen2.5-Math-7B, 5-benchmark average | pass@1 accuracy: 49.3 -> 53.3 | Helped |
| Add verifiable rewards to min-form process-reward training | PURE-PRM · Qwen2.5-Math-7B, AMC23 | pass@1 accuracy: 60.0 -> 82.5 | Helped |
| Add verifiable rewards to min-form process-reward training | PURE-PRM · Qwen2.5-7B, 5-benchmark average | pass@1 accuracy: 45.6 -> 47.6 | Helped |
| Use process rewards alongside verifiable rewards instead of verifiable rewards alone | PURE-VR · Qwen2.5-Math-7B, 5-benchmark average | pass@1 accuracy: 48.3 -> 53.3 | Helped |
| Use process rewards instead of verifiable rewards | PURE-VR · Qwen2.5-Math-7B, 5-benchmark average | pass@1 accuracy: 48.3 -> 49.3 | Helped |
| Use PRM-involved training rather than sparse-reward PURE-VR | PURE-PRM and PURE-PRM+VR · MATH-500 training | training steps to reach the same accuracy: around 30% | Helped |
| Add verifiable rewards to process-reward training | PURE-PRM · training reward hacking | reward hacking: null | Helped |
| Set transform temperature to 0.1 rather than 0.01 (ablation) | PURE-PRM+VR · Qwen2.5-7B, 5-benchmark average | pass@1 accuracy: 42.8 -> 45.6 | Helped |
| Set transform temperature to 0.1 rather than 1.0 (ablation) | PURE-PRM+VR · Qwen2.5-7B, 5-benchmark average | pass@1 accuracy: 42.8 -> 45.6 | Helped |
| Replace the token-level process-reward baseline with a step-level baseline (ablation) | PURE process-reward advantage estimator · reasoning responses | number of generated steps | Hurt |
Surprising findings
- Summation-form credit assignment collapses training at around step 25 even though the same PRM works in test-time scaling.
- Pseudo-positive long, repetitive responses judged correct by the verifier cause training collapse within about 5 gradient steps.
Limitations
- Reward hacking still occurs with PRM-only training (case 3, zero-step responses) and is only delayed, not eliminated, by min-form credit assignment.
- Pseudo-positive repetitive samples evade both verifiers and the discriminative PRM; the LCP repetition detector is easily bypassed.
- Results are from single runs with greedy decoding and no error bars; evaluation limited to math benchmarks and three Qwen base models.
Experimental setup
Datasets: PRM800K MATH SimpleRL RFT dataset GSM8K ProcessBench PRMBench AIME24 AMC23 MATH500 Minerva Math OlympiadBench
Benchmarks: MATH500 Minerva Math OlympiadBench AIME24 AMC23 GSM8K ProcessBench PRMBench
Metrics: pass@1 accuracy (greedy) average accuracy best-of-N accuracy F1
Baselines: Eurus-2-7B-PRIME SimpleRL-Zoo Qwen2.5-7B-DPO-Zero Deepseek-PRM-7B Skywork-PRM-7B Math-Shepherd-PRM-7B RLHFlow-PRM-Deepseek-8B Qwen2.5-Math-7B-PRM800K
Builds on: RLOO REINFORCE++ GRPO PRIME SimpleRL-Zoo
Closest papers by meaning
- Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.90)
- Discriminative Policy Optimization for Token-Level Reward Models (ICML 2025 · similarity 0.89)
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.89)
- Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.89)
- Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs (NeurIPS 2025 · similarity 0.89)
- Free Process Rewards without Process Labels (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.