Research map / Large language models / Reasoning and chain of thought
Step-GRPO: Enhancing Reasoning Quality and Efficiency via Structured PRM-Based Reinforcement Learning
Weijie Li, Jin Wang, Liang-Chih Yu, Xuejie Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Step-GRPO, an RL method for large reasoning models that uses a structured process reward model over reasoning steps to improve reasoning quality and reduce overthinking.
Problem
Large reasoning models overthink, incurring high test-time compute; outcome-level rewards favor short correct paths but ignore intermediate reasoning quality, while dense step-level rewards are the alternative.
Approach
Replaces purely outcome-level rewards with structured process-reward-model (PRM) signals over reasoning steps within a GRPO-style RL framework, aiming to reward reasoning quality and efficiency jointly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train DeepSeek-R1-Distill-1.5B with Step-GRPO | R1-Distill-1.5B · AIME'24 | pass@1: 28.7 -> 36.7 | Improved |
| train DeepSeek-R1-Distill-1.5B with Step-GRPO | R1-Distill-1.5B · AIME'24 | avg tokens: 11 815 -> 2 779 | Improved |
| train DeepSeek-R1-Distill-1.5B with Step-GRPO | R1-Distill-1.5B · AIME'24 | PRM score: 0.24 -> 0.31 | Improved |
| train Qwen2.5-Math-1.5B-Base-style model with Step-GRPO (Step-RS1) vs LUFFY | Qwen2.5-Math-1.5B · AIME'24 | pass@1: 16.7 -> 20.0 | Improved |
| step-attention aggregation in place of static min/mean/max PRM aggregation (ablation) | Step-GRPO (Qwen backbone) · MATH500 and AIME'24 | accuracy, PRM score, tokens | Improved |
| add positional bias to step-attention (ablation) | Step-GRPO step-attention · MATH500 and AIME'24 | accuracy, PRM score, tokens | Improved |
| use max aggregation of PRM step scores (ablation) | Step-GRPO · training accuracy (Qwen backbone) | accuracy | Hurt |
| lower sampling temperature from 0.5/0.7 to 0.3 (ablation) | Step-GRPO early training · AIME'24 (step 1.5K) | accuracy: 0.32/0.30 -> 0.36 | Improved |
| use PRM-based step-attention reward instead of cosine rule-based reward (ablation) | Step-GRPO · AIME'24 and MATH500 | accuracy and token cost | Improved |
| Step-RS2 versus DeepScaleR on AIME'25 | R1-Distill-1.5B · AIME'25 | pass@1: 26.7 -> 26.7 | No clear effect |
Experimental setup
Baselines: outcome-level reward RL methods
Builds on: GRPO process reward model
Closest papers by meaning
- S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models (NeurIPS 2025 · similarity 0.90)
- Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning (AAAI 2026 · similarity 0.89)
- Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.89)
- TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback (NeurIPS 2026 · similarity 0.89)
- DAPO : Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization (NeurIPS 2025 · similarity 0.89)
- Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
