Research map / Large language models / Agents, code and math
AlphaMath Almost Zero: Process Supervision without Process
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AlphaMath, an iterative MCTS-based self-training framework that uses only question-answer pairs (no process annotations) to train an LLM policy with an appended value head, plus a step-level beam search for inference.
Problem
Improving mathematical reasoning of LLMs without costly human- or GPT-4-annotated process-supervised solutions.
Approach
Integrate a value head into the LLM policy and use MCTS over the LLM to generate correct and incorrect step-level solutions from only final answers; the value head is trained on MCTS-derived state values, and a step-level beam search uses the value model to select steps instead of relying on prior probabilities.
Key results
- DeepSeekMath-Base-7B with AlphaMath (K=3) and step-level beam search (B1=3) reaches 84.1 on GSM8K and 66.3 on MATH without process annotations, comparable to or above 7B SFT baselines.
- Step-level beam search (B1=1) improves over greedy decoding from 53.6 to 62.8 on MATH, and to 81.1 on GSM8K.
- On Llama3-base, AlphaMath adds roughly +20 points average improvement on in-domain and OOD benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply AlphaMath iterative training instead of using the 2-shot prompt alone | DeepSeekMath-Base-7B · GSM8K | accuracy: 59.7 -> 73.5 | Helped |
| Apply AlphaMath iterative training instead of using the 2-shot prompt alone | DeepSeekMath-Base-7B · MATH | accuracy: 33.2 -> 53.6 | Helped |
| Apply AlphaMath iterative training instead of using the 2-shot prompt alone | DeepSeekMath-Base-7B · GaoKao2023 | accuracy: 21.9 -> 40.5 | Helped |
| Apply AlphaMath iterative training instead of using the 2-shot prompt alone | DeepSeekMath-Base-7B · OCWCourses | accuracy: 9.2 -> 26.1 | Helped |
| Add step-level beam search with B1=1 (ablation) | AlphaMath, DeepSeekMath-Base-7B · GSM8K | accuracy: 73.5 -> 81.1 | Helped |
| Increase SBS beam size from B1=1 to B1=3 (ablation) | AlphaMath, DeepSeekMath-Base-7B · MATH | accuracy: 62.8 -> 66.3 | Helped |
| Increase SBS beam size from B1=1 to B1=3 (ablation) | AlphaMath, DeepSeekMath-Base-7B · GaoKao2023 | accuracy: 46.2 -> 51.4 | Helped |
| Increase SBS beam size from B1=1 to B1=3 (ablation) | AlphaMath, DeepSeekMath-Base-7B · OCWCourses | accuracy: 30.5 -> 33.1 | Helped |
| Use MCTS inference instead of SBS with B1=1 (ablation) | AlphaMath, DeepSeekMath-Base-7B · MATH | accuracy: 62.8 -> 64.0 | Helped |
| Use MCTS inference instead of SBS with B1=1 (ablation) | AlphaMath, DeepSeekMath-Base-7B · OCWCourses | accuracy: 30.5 -> 33.8 | Helped |
| Increase SBS beam size from B1=3 to B1=5 (ablation) | AlphaMath, DeepSeekMath-Base-7B · MATH | accuracy: 66.30 -> 65.98 | Hurt |
| Apply AlphaMath iterative training | Llama3-base · GSM8K | accuracy: 40.7 -> 59.4 | Helped |
Surprising findings
- Incorrect solutions often contain correct intermediate steps, so intermediate Q-values in incorrect solutions can be updated toward 1 via backup.
Limitations
- Reward still relies on ground-truth final answers, so it is not truly from zero.
- Training uses a fixed set of only 15k question-answer pairs across iterations.
- MCTS inference is computationally expensive compared to step-level beam search.
Experimental setup
Datasets: GSM8K MATH GaoKao2023 OCWCourses MARIO 27k data
Benchmarks: GSM8K MATH GaoKao2023 OCWCourses
Metrics: accuracy
Baselines: GPT-4 GPT-4 (PAL) ChatGPT Gemini-1.5 Pro Claude-3.5-Sonnet MAmmoTH-Coder MathCoder ToRA MARIO MathGenie DeepSeekMath-Instruct
Builds on: AlphaGo Zero PUCT ReAct Program-aided language models DeepSeekMath
Closest papers by meaning
- SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling (AAAI 2026 · similarity 0.89)
- Improving Value-based Process Verifier via Low-Cost Variance Reduction (AAAI 2026 · similarity 0.89)
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.88)
- Unlocking Multimodal Mathematical Reasoning via Process Reward Model (NeurIPS 2025 · similarity 0.87)
- StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models (ICLR 2026 · similarity 0.87)
- GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.