Research map / Large language models / Reasoning and chain of thought

AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence

ICML 2025 · Large language models › Reasoning and chain of thought · approach: Process reward models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdaptiveStep, which splits LLM reasoning responses into steps at low-confidence tokens, and uses the resulting automatically labeled data to train process reward models (ASPRM) for math and code, applied in Best-of-N and token-level value-guided decoding.

Problem

Process reward model training relies on rule-based step division (line breaks, fixed token counts), which yields coarse steps that miss true decision points, and manual step annotation is costly.

Approach

Divide reasoning responses into steps at tokens where the policy model's sampled-token confidence falls below a data-driven threshold (set by a proportion of tokens), then label steps via rollouts with hard estimation, requiring no manual annotation; the PRM also serves as a value function for token-level guided decoding.

Key results

• ASPRM-L with token-level value-guided decoding improves greedy accuracy on GSM8k from 81.80 to 83.47 and on MATH500 from 38.80 to 42.00 for MetaMath-L.
• ASPRM-D with TVD improves LeetCodeDataset Pass@1 from 26.28 to 28.00 and LiveCodeBench from 19.21 to 19.92.
• Paper claims data construction cost under 70% of the open-source baselines for math PRM training data.

Measured effects reported in the paper

Surprising findings

• Only 2.7% of decision tokens are newline tokens, indicating line breaks are not where the model is most uncertain.
• Training the math PRM on data from a different generator (MetaMath-Llama) retains judging ability but performs weaker than in-distribution data, indicating limited transferability.

Limitations

• Transferability across data generators is limited; a PRM trained on another model's data performs worse.
• Performance on MATH500 trails ER-PRM and Math-Shepherd in some settings, and ASPRM-D struggles on GSM8k in cross-domain tests.

Experimental setup

Datasets: MetaMathQA GSM8k train MATH train LeetCodeDataset GSM-Symbolic LiveCodeBench

Benchmarks: GSM8k MATH500 LeetCodeDataset LiveCodeBench GSM-Symbolic

Metrics: Best-of-N accuracy Pass@1 Accuracy@1 token-level value-guided decoding accuracy

Baselines: Math-Shepherd ER-PRM Code ORM (ORM-D) greedy decoding

Builds on: Math-Shepherd MetaMath Monte Carlo rollout labeling

Closest papers by meaning

• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.92)
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.90)
• Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.90)
• Free Process Rewards without Process Labels (ICML 2025 · similarity 0.89)
• VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data (ICML 2025 · similarity 0.89)
• Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning (AAAI 2026 · similarity 0.89)