Research map / Large language models / Reasoning and chain of thought

Promoting Efficient Reasoning with Verifiable Stepwise Reward

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VSRM, a rule-based verifiable stepwise reward that segments reasoning traces at special tokens, scores each prefix by sub-rollout accuracy differences with decayed propagation, and combines it with PPO or Reinforce++ to cut overthinking in distilled reasoning models.

Problem

Large reasoning models trained with outcome-only RLVR overthink, spending excessive tokens on simple problems; existing efficiency methods rely on preset token budgets or difficulty estimates.

Approach

Segment the reasoning trace at rule-based step tokens (e.g., "however", "wait"), estimate each prefix's solve accuracy via sampled continuations, and reward the accuracy change between adjacent prefixes with a discounted lookahead, giving dense stepwise rewards without a trained PRM.

Key results

• On DS-Distill-1.5B, VSRM-PPO reduces average-style length substantially on AIME24 (12605 to 7065 tokens) while raising pass@1 from 28.8 to 29.2.
• On DS-Distill-7B, VSRM-R++ cuts AIME24 length from 10529 to 6773 tokens with pass@1 of 52.2 vs 53.1 base.
• Ablation on DS-Distill-1.5B shows removing propagation decay drops length compression (AIME24 8638 vs 7065 tokens with full VSRM-PPO).

Surprising findings

• Plain PPO without stepwise reward already reduces length somewhat, attributed to format constraints.
• Adding a length-penalized outcome reward compressed less effectively than VSRM alone.

Limitations

• Overthinking detection relies on DeepSeek-R1 prompting, which the authors say is only a rough evaluation and not precise.
• Step-level accuracy estimation requires multiple extra rollouts per sub-trajectory, adding training cost.
• Evaluation focused on mathematical reasoning with mostly distilled base models.

Experimental setup

Datasets: DeepScaleR training dataset MATH-500

Benchmarks: AIME24 AIME25 MATH-500 AMC23 Minerva OlympiadBench

Metrics: pass@1 output length overthinking detection count pass@k

Baselines: LC-R1 ThinkPrune-2k AdaptThink Efficient Reasoning GRPO-LEAD

Builds on: PPO Reinforce++ GRPO VeRL DeepScaleR

Closest papers by meaning

• How Far Are We from Optimal Reasoning Efficiency? (NeurIPS 2025 · similarity 0.91)
• Learn to Reason Efficiently with Adaptive Length-based Reward Shaping (ICLR 2026 · similarity 0.90)
• Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.90)
• DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains (ICLR 2026 · similarity 0.90)
• Self-Aligned Reward: Towards Effective and Efficient Reasoners (ICLR 2026 · similarity 0.90)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.90)