Research map / Large language models / Reasoning and chain of thought
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows small LMs (1.5B-7B) can reach o1-level math reasoning by self-evolving a policy SLM and a process preference model (PPM) through code-augmented MCTS rollouts, without distillation from stronger models.
Problem
Training strong math policy models and reliable process reward models for test-time System 2 reasoning is hard because high-quality step-level data is scarce and distillation from teacher LLMs plateaus at the teacher's capability.
Approach
Generate step-by-step verified trajectories via MCTS where each step is a code-plus-comment candidate kept only if its Python executes, assign Q-values by rollouts, train a PPM with pairwise Bradley-Terry preference loss on Q-value-selected step pairs instead of regressing noisy Q-values, and iterate this loop for four self-evolution rounds.
Key results
- On MATH, rStar-Math with Qwen2.5-Math-7B reaches 90.0% with 64 trajectories (vs. 58.8% base), matching o1-mini and above o1-preview's 85.5%.
- On AIME 2024, rStar-Math (7B, 64 trajectories) solves 53.3% (8/15 average), versus 44.6% for o1-preview.
- PPM-guided MCTS outperforms ORM Best-of-N (MATH 89.4 vs 82.6 with same policy) and Q-value PRM (PQM 88.2 MATH), showing preference-based PRM training helps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase test-time search to 64 trajectories | rStar-Math with Qwen2.5-Math-7B · MATH | Pass@1 accuracy: 88.4 -> 90.0 | Helped |
| Increase test-time search to 64 trajectories | rStar-Math with Qwen2.5-Math-7B · AIME 2024 | Pass@1 accuracy: 50.0 -> 53.3 | Helped |
| Apply rStar-Math deep thinking | Qwen2.5-Math-7B · MATH | Pass@1 accuracy: 58.8 -> 90.0 | Helped |
| Apply rStar-Math deep thinking | Qwen2.5-Math-7B · College Math | Pass@1 accuracy: 41.6 -> 60.5 | Helped |
| Apply rStar-Math deep thinking | Qwen2.5-Math-7B · AIME 2024 | Pass@1 accuracy: 0.0 -> 53.3 | Helped |
| Train on step-by-step verified trajectories rather than random samples (ablation) | Qwen2.5-Math-7B SFT · MATH | accuracy: 72.4 -> 78.4 | Helped |
| Train on step-by-step verified trajectories rather than rejection-sampled solutions (ablation) | Qwen2.5-Math-7B SFT · MATH | accuracy: 73.4 -> 78.4 | Helped |
| Train on step-by-step verified trajectories rather than random samples (ablation) | Qwen2.5-Math-7B SFT · AIME | accuracy: 10.0 -> 26.7 | Helped |
| Train on step-by-step verified trajectories rather than rejection-sampled solutions (ablation) | Qwen2.5-Math-7B SFT · AIME | accuracy: 13.3 -> 26.7 | Helped |
| Use pairwise preference training instead of Q-value scores as reward labels (ablation) | Qwen2.5-Math-7B policy with MCTS · MATH | accuracy: 88.2 -> 89.4 | Helped |
| Use step-level PPM/MCTS instead of ORM with Best-of-N (ablation) | Qwen2.5-Math-7B policy · MATH | accuracy: 82.6 -> 89.4 | Helped |
| Progress through self-evolution rounds (ablation) | rStar-Math with Qwen2.5-Math-7B · AIME 2024 | Pass@1 accuracy: 10.0 -> 50.0 | Helped |
Surprising findings
- Self-reflection-like backtracking emerged in MCTS deep thinking without any self-reflection training data or prompts.
- Round-1 policy SFT on DeepSeek-Coder-V2 trajectories was followed by round-2 gains driven mostly by the PPM, with the final reasoning accuracy converging across policy sizes once the PPM is applied.
Limitations
- Focuses on word problems; geometry requiring visual understanding is unsupported (8 of the unsolved AIME problems were geometry).
- Relies on GPT-4 synthesized problems that are sometimes mislabeled; the remaining unsolved synthetic problems were found to be largely incorrect.
- Requires code-executable reasoning, so generalization to other domains needs a verification mechanism.
Experimental setup
Datasets: NuminaMath MetaMath MATH train AMC-AIME train GSM8K
Benchmarks: MATH AIME 2024 AMC 2023 Olympiad Bench College Math GSM8K Gaokao En 2023
Metrics: pass@1 accuracy pass@N accuracy
Baselines: OpenAI o1-preview OpenAI o1-mini GPT-4o Qwen2.5-Math-72B-Instruct + 72B ORM PQM (Q-value PRM) ORM DeepSeek-Coder-V2-Instruct Claude 3.5 Sonnet
Builds on: AlphaGo rStar Bradley-Terry pairwise ranking loss UCT
Closest papers by meaning
- Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solver (ICLR 2025 · similarity 0.90)
- JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models (NeurIPS 2024 · similarity 0.89)
- Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning (AAAI 2024 · similarity 0.88)
- Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability (ICML 2026 · similarity 0.87)
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data (ICLR 2025 · similarity 0.87)
- MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.