Research map / Large language models / Agents, code and math
What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Ablates natural-language thought descriptions from MCTS-collected step-level preference data to show that math-expression-only step-level reward models perform comparably to full-context ones, while NL-only models struggle.
Problem
Step-level reward models (SRMs) trained on MCTS-collected step preferences improve math reasoning, but what they actually reward, and whether natural-language thought descriptions are necessary, is unclear.
Approach
Decomposes each reasoning step into natural-language thoughts and math expressions, trains SRM variants with different input structures (full-context, math-only, single-step math-only, next-thought), and compares them as greedy-search scorers to test what drives step-level reward modeling.
Key results
- Math-Only SRM (DeepSeek-Math-7B-Base) reaches 85.82% on GSM8K and 39.64% on MATH under greedy search, versus 86.20% and 38.58% for the Full-Context SRM; t-tests show no significant difference (p > 0.4).
- Next-Thought SRM gives only +0.91% on GSM8K and -0.18% on MATH with DeepSeek-Math-7B-Base, indicating NL-based preference learning is hard.
- Math-Only SRM (trained on Llama-3-8B-Instruct MCTS data) lifts Llama-3-70B-Instruct from 90.37% to 92.95% on GSM8K and 48.48% to 54.12% on MATH.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove thought descriptions from SRM input (ablation) | Full-Context SRM + greedy search · GSM8K and MATH test | accuracy: GSM8K: 86.20 (+7.73) -> 85.82 (+7.35); MATH: 38.58 (+7.42) -> 39.64 (+8.48) | No clear effect |
| Use the full sequence of math expressions rather than only the current expression (ablation) | Math-Only SRM + greedy search · GSM8K and MATH test | accuracy: GSM8K: +3.64% -> +7.35%; +3.18% -> +5.31%; MATH: 6.30% -> 8.48%; +1.92% -> +3.94% | Helped |
| Score next thoughts without next math expressions (ablation) | Full-Context SRM + greedy search · GSM8K and MATH test | accuracy: GSM8K: 86.20 (+7.73) -> 79.38 (+0.91); 82.94 (+4.47) -> 81.73 (+3.26); MATH: 38.58 (+7.42) -> 30.98 (-0.18); 35.58 (+4.4 | Hurt |
| Use Math-Only SRM to guide a larger model | Llama-3-70B-Instruct · GSM8K and MATH test | accuracy: GSM8K: 90.37 -> 92.95 (+2.58); MATH: 48.48 -> 54.12 (+5.64) | Helped |
Surprising findings
- Removing natural-language thought descriptions from SRM inputs has minimal effect, and math-only SRMs sometimes outperform full-context SRMs (e.g., MATH 39.64 vs 38.58).
- Next-thought (natural-language) SRMs struggle to learn preferences and give almost no gain over the baseline.
Limitations
- Preference alignment and policy updates are not included; SRMs are evaluated only as greedy-search scorers.
- Results rely on a single MCTS data generator (Llama-3-8B-Instruct) and a small set of base models.
- Explanation of why Llama-3-8B underperforms as an SRM base is left open.
Experimental setup
Datasets: GSM8K MATH
Benchmarks: GSM8K MATH
Metrics: accuracy accuracy gain over baseline preference prediction accuracy t-test p-value
Baselines: Pass@1 (3-shot) Llama-3-8B-Instruct
Builds on: Math-Shepherd Monte Carlo Tree Search AlphaZero Tree of Thoughts
Closest papers by meaning
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.88)
- AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence (ICML 2025 · similarity 0.88)
- Unlocking Multimodal Mathematical Reasoning via Process Reward Model (NeurIPS 2025 · similarity 0.87)
- RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.87)
- rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (ICML 2025 · similarity 0.87)
- Free Process Rewards without Process Labels (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.