Research map / Large language models / Reasoning and chain of thought
ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReST-MCTS*, which uses process-reward-guided MCTS* to infer per-step value labels from oracle answers, then mutually self-trains the policy and process reward model over iterations.
Problem
LLM self-training usually keeps any trace with a correct final answer, so intermediate reasoning is often wrong; training step-level process reward models normally needs costly per-step human annotation.
Approach
Tree search (MCTS*) guided by a learned process value model; per-step quality values are inferred automatically from rollouts that reach the oracle correct answer (reasoning distance and hard estimation). These values train the PRM, and the searched correct traces train the policy, iterating mutually.
Key results
- After 2 iterations, ReST-MCTS* reaches 29.02 average on LLaMA-3-8B-Instruct versus 27.73 for Self-Rewarding and 26.83 for ReST-EM (2nd iteration).
- On MATH500 verification with 256 outputs, SC + ReST-MCTS* (Value) reaches 39.0 vs 38.3 for SC+MATH-SHEPHERD on Mistral-7B: MetaMATH.
- On SciBench, ReST-MCTS* achieves 16.77 average with GLM4 vs 15.82 for ToT and 12.68 for CoT; with GPT-3.5-turbo 10.06 vs 8.44 ToT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReST-MCTS* self-training instead of ReST^EM or Self-Rewarding | LLaMA-3-8B-Instruct · Average across MATH, GPQA Diamond, and CEval-Hard, 2nd iteration | accuracy: 29.02; 27.73 | Helped |
| Use ReST-MCTS* self-training instead of ReST^EM or Self-Rewarding | Mistral-7B: MetaMATH · Average across MATH, GPQA Diamond, and CEval-Hard, 2nd iteration | accuracy: 26.50; 25.22 | Helped |
| Use ReST-MCTS* self-training instead of ReST^EM or Self-Rewarding | SciGLM-6B · Average across MATH, GPQA Diamond, and CEval-Hard, 2nd iteration | accuracy: 35.90; 33.56 | Helped |
| Use ReST-MCTS* value model for Self-Consistency verification instead of MATH-SHEPHERD | Mistral-7B: MetaMATH · GSM8K test | accuracy: 87.1; 87.5 | Helped |
| Use ReST-MCTS* value model for Self-Consistency verification instead of SC + MS | Mistral-7B: MetaMATH · MATH500 | accuracy: 38.3; 39.0 | Helped |
| Use process-reward-guided MCTS* instead of ToT | GLM4 · SciBench, All (Ave.) | accuracy: 15.82; 16.77 | Helped |
| Use process-reward-guided MCTS* instead of ToT | GPT-3.5-turbo · SciBench, All (Ave.) | accuracy: 8.44; 10.06 | Helped |
| Use process-reward-guided MCTS* instead of ToT | LLaMA2-13B-Chat · SciBench, All (Ave.) | accuracy: 2.37; 2.90 | Helped |
| Use ReST-MCTS* tree search instead of Self-Consistency | ReST-MCTS* search policy · MATH test, after more self-training iterations | accuracy: 42.5%; 48.5% | Helped |
Surprising findings
- ToT sometimes surpasses ReST-MCTS* on individual SciBench subjects.
- For the small LLaMA2-13B-Chat policy, the gains from tree search are not prominent, suggesting small policies struggle with complex tree search.
Limitations
- Evaluated mainly on math and science reasoning; generalization to coding, agents, or tasks without ground truth is untested.
- Value model is small-scale and data filtering techniques need improvement, as the authors note.
- Self-training experiments mainly use math data, so transfer to science benchmarks is not studied.
Experimental setup
Datasets: SciInstruct MATH train GSM8K train TheoremQA
Benchmarks: MATH GPQA Diamond CEval-Hard GSM8K MATH500 SciBench SciEval
Metrics: accuracy
Baselines: CoT Tree-of-Thought Self-Consistency Best-of-N ReST-EM Self-Rewarding LM MATH-SHEPHERD ORM
Builds on: ReST-EM MATH-SHEPHERD TS-LLM Tree-of-Thought MuZero
Closest papers by meaning
- ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search (NeurIPS 2026 · similarity 0.91)
- Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.90)
- Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.89)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
- Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.89)
- Policy Guided Tree Search for Enhanced LLM Reasoning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.