Research map / Large language models / Agents, code and math
Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AlphaLLM, an imagination-searching-criticizing loop that pairs option-level MCTS with value, process, and outcome critics to generate self-training data for LLMs on math reasoning without extra annotations.
Problem
LLMs struggle with complex reasoning and planning, and self-improvement via self-critique is unreliable; it is unclear how to combine MCTS with LLMs for self-improvement given limited data, huge token search spaces, and subjective feedback.
Approach
Combines prompt synthesis, option-level MCTS (multi-token options as search nodes, importance-based adaptive branching, state merging, fast rollouts) and three critics (value function, process reward model, outcome reward model with tool use) so that MCTS trajectories with high critic scores are used to finetune the policy in an iterative loop.
Key results
- AlphaLLM with option-level MCTS raises Llama-2-70b from 57.8 to 92.0 on GSM8K after two self-improvement rounds, matching GPT-4's reported 92.0.
- On MATH, WizardMath-70B-V1.0 improves from 20.7 to 51.0 with MCTS decoding after one round of self-improvement.
- Ablation on GSM8K shows vanilla MCTS at 79.5 rising to 88.9 with all components including large rollout count.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply synthetic-prompt self-improvement to the policy | Llama-2-70B · GSM8K | accuracy: 57.8 -> 73.7 | Helped |
| Decode the self-improved policy with EMCTS | AlphaLLM · GSM8K | accuracy: 73.7 -> 92.0 | Helped |
| Apply synthetic-prompt self-improvement to the policy | WizardMath-70B-V1.0 · MATH | accuracy: 20.7 -> 23.6 | Helped |
| Decode the self-improved policy with EMCTS | AlphaLLM · MATH | accuracy: 23.6 -> 51.0 | Helped |
| Add adaptive branching (ablation) | Vanilla MCTS · GSM8K | accuracy: 79.5 -> 84.9 | Helped |
| Add PRM (ablation) | EMCTS with adaptive branching · GSM8K | accuracy: 84.9 -> 85.9 | Helped |
| Add fast-rollout ORM (ablation) | EMCTS with adaptive branching and PRM · GSM8K | accuracy: 85.9 -> 86.5 | Helped |
| Add state merging (ablation) | EMCTS with fast-rollout ORM · GSM8K | accuracy: 86.5 -> 87.0 | Helped |
| Increase the number of rollouts (ablation) | EMCTS with other components · GSM8K | accuracy: 87.0 -> 88.9 | Helped |
| Add tool augmentation to ORM (ablation) | MATH EMCTS · MATH | accuracy: 38.8 -> 44.1 | Helped |
| Use option-level rather than sentence-level search (ablation) | MATH EMCTS · MATH | accuracy: 44.1 -> 45.4 | Helped |
| Vary the state-merge heuristic or model (ablation) | EMCTS · GSM8K | accuracy: 86.3; 86.7; 86.8; 87.0 | No clear effect |
Surprising findings
- Greedy decoding of the self-improved model improves only modestly (73.7 on GSM8K, 23.6 on MATH), far below MCTS-decoded performance.
- The process reward model has high recall but poor calibration (ECE 0.375) compared with the value function (ECE 0.032).
Limitations
- Evaluated only on mathematical reasoning (GSM8K and MATH), so generality to other domains is untested.
- Synthetic prompt generation is simple and the critic models stay static during self-improvement.
- Greedy decoding after self-improvement is much weaker than MCTS decoding, suggesting the loop does not fully transfer gains to the policy.
Experimental setup
Datasets: GSM8K MATH
Benchmarks: GSM8K MATH
Metrics: accuracy average rollouts (number of tree nodes)
Baselines: GPT-4 GPT-3.5 Gemini 1.5 Pro Claude-2 PaLM-2 WizardMath-70B-V1.0 Llama-2-70b SFT self-consistency re-ranking
Builds on: AlphaGo Monte Carlo Tree Search options framework Abel
Closest papers by meaning
- Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs (NeurIPS 2025 · similarity 0.91)
- Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.91)
- ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement (ICLR 2025 · similarity 0.90)
- Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024 · similarity 0.90)
- Self-Improvement in Language Models: The Sharpening Mechanism (ICLR 2025 · similarity 0.90)
- On the self-verification limitations of large language models on reasoning and planning tasks (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.