Research map / Large language models / Reasoning and chain of thought

ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ReST-RL, combining ReST-style prefix-based GRPO training (ReST-GRPO) with a value model trained on self-collected MCTS targets and used for value-guided MCTS decoding (VM-MCTS), evaluated mainly on coding benchmarks.

Problem

GRPO yields low-information updates when sampled responses in a group receive similar rewards, and process reward models require costly step-level annotation for reasoning, especially coding.

Approach

ReST-GRPO filters prompts by reward variance and restarts online GRPO from prefixes of high-reward solutions (sampling fresh suffixes rather than imitating them); a value model is then trained on MCTS-collected expected-reward targets under the fixed policy and used both for MCTS allocation and Best-of-N selection.

Key results

• ReST-GRPO after 2 iterations on Qwen3-8B reaches 0.655 average versus 0.590 for GRPO and 0.597 for DAPO over the six coding benchmarks.
• VM-MCTS with ReST-GRPO policy reaches 0.642 on APPS-500 versus 0.565 policy-only, and the full ReST-RL reaches 0.689 average on Qwen3-8B versus 0.538 for ORM-MCTS.
• Cross-domain transfer without target tuning: Qwen3-8B on MATH rises from 0.780 to 0.872 and GPQA-Diamond from 0.449 to 0.490 with the code-trained pipeline.

Surprising findings

• Ablation shows the exponential prefix sampling only modestly beats uniform prefix sampling and naive GRPO gains are small relative to full method (0.180 to 0.207 on APPS-500 in the controlled setting).
• Some ReST-GRPO gains are small on certain models, e.g. DS-Coder-6.7B average improves only 0.493 to 0.511 after one iteration.

Limitations

• Primary evaluation is coding; math and science transfer is preliminary and uses only a single base model.
• Value model targets are policy-dependent, so VMs must be retrained for each policy.
• Full pipeline is sequential rather than jointly optimized, and ReST-GRPO gains are limited for some base models.

Experimental setup

Datasets: BigCodeBench DS-1000 APPS

Benchmarks: HumanEval HumanEval+ MBPP MBPP+ APPS-500 BigCodeBench MATH Omni-MATH GPQA-Diamond

Metrics: pass@1 fraction of passed test cases (APPS)

Baselines: GRPO DAPO ReST-DPO ORM PRM ORM-MCTS PRM-BoN E2H-C

Builds on: GRPO ReST ReST-EM Math-Shepherd PPO-MCTS DAPO

Closest papers by meaning

• Tricks or Traps? A Deep Dive into RL for LLM Reasoning (ICLR 2026 · similarity 0.92)
• ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search (NeurIPS 2024 · similarity 0.91)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.91)
• TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning (ICLR 2026 · similarity 0.90)
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.90)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)