AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Step-GRPO: Enhancing Reasoning Quality and Efficiency via Structured PRM-Based Reinforcement Learning

Weijie Li, Jin Wang, Liang-Chih Yu, Xuejie Zhang

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Step-GRPO, an RL method for large reasoning models that uses a structured process reward model over reasoning steps to improve reasoning quality and reduce overthinking.

Problem

Large reasoning models overthink, incurring high test-time compute; outcome-level rewards favor short correct paths but ignore intermediate reasoning quality, while dense step-level rewards are the alternative.

Approach

Replaces purely outcome-level rewards with structured process-reward-model (PRM) signals over reasoning steps within a GRPO-style RL framework, aiming to reward reasoning quality and efficiency jointly.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train DeepSeek-R1-Distill-1.5B with Step-GRPOR1-Distill-1.5B · AIME'24pass@1: 28.7 -> 36.7Improved
train DeepSeek-R1-Distill-1.5B with Step-GRPOR1-Distill-1.5B · AIME'24avg tokens: 11 815 -> 2 779Improved
train DeepSeek-R1-Distill-1.5B with Step-GRPOR1-Distill-1.5B · AIME'24PRM score: 0.24 -> 0.31Improved
train Qwen2.5-Math-1.5B-Base-style model with Step-GRPO (Step-RS1) vs LUFFYQwen2.5-Math-1.5B · AIME'24pass@1: 16.7 -> 20.0Improved
step-attention aggregation in place of static min/mean/max PRM aggregation (ablation)Step-GRPO (Qwen backbone) · MATH500 and AIME'24accuracy, PRM score, tokensImproved
add positional bias to step-attention (ablation)Step-GRPO step-attention · MATH500 and AIME'24accuracy, PRM score, tokensImproved
use max aggregation of PRM step scores (ablation)Step-GRPO · training accuracy (Qwen backbone)accuracyHurt
lower sampling temperature from 0.5/0.7 to 0.3 (ablation)Step-GRPO early training · AIME'24 (step 1.5K)accuracy: 0.32/0.30 -> 0.36Improved
use PRM-based step-attention reward instead of cosine rule-based reward (ablation)Step-GRPO · AIME'24 and MATH500accuracy and token costImproved
Step-RS2 versus DeepScaleR on AIME'25R1-Distill-1.5B · AIME'25pass@1: 26.7 -> 26.7No clear effect

Experimental setup

Baselines: outcome-level reward RL methods

Builds on: GRPO process reward model

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy