Research map / Large language models / Reasoning and chain of thought
Value-Guided Search for Efficient Chain-of-Thought Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a 1.5B token-level value model on 2.5M roll-in/roll-out reasoning traces with cross-entropy regression-via-classification, then uses block-wise beam search guided by it (with final weighted majority vote) to improve test-time scaling of DeepSeek-R1-Distill models on competition math.
Problem
Process reward models for guiding search require a pre-defined notion of step and expensive step-level labels, which are hard to define and collect for long chain-of-thought reasoning models.
Approach
Train a token-level value model via classification on random roll-in/roll-out pairs (no step definition or step labels), and perform block-wise beam search guided by its values, aggregating final beams with a weighted majority vote instead of best-of-N.
Key results
- VGS with DeepSeek-VM-1.5B on DeepSeek-1.5B (N=256) reaches 39.8% average on AIME-25 and HMMT-25, vs 31.6% for MV@256 and 37.0% for WMV.
- VGS with DeepSeek-VM-1.5B guiding DeepSeek-14B (N=64) reaches 61.0% average over AIME-24/25 and HMMT-24/25, versus 59.6% for MV, and is reported as on par with DeepSeek-R1 (671B).
- VGS with the 1.5B value model outperforms 7B PRMs (Math-Shepherd, Qwen2.5-Math-PRM) when used for search on the same generator.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use block-wise VGS instead of WMV alone | DeepSeek-1.5B with DeepSeek-VM-1.5B · AIME-25 and HMMT-25 AVG | accuracy: 43.2±1.4 -> 45.7±1.0 | Helped |
| use VGS instead of majority voting | DeepSeek-7B · AIME-25 and HMMT-25 AVG | accuracy: 50.5±0.9 -> 56.4±0.8 | Helped |
| use VGS instead of majority voting | DeepSeek-14B · AIME-25 and HMMT-25 AVG | accuracy: 59.6±0.9 -> 61.0±0.9 | Helped |
| use DeepSeek-VM-1.5B rather than the 7B PRM scorers for VGS | DeepSeek-1.5B · AIME-25 and HMMT-25 AVG | accuracy: 40.2±0.5 -> 45.7±1.0; 41.6±0.9 -> 45.7±1.0 | Helped |
| use weighted majority voting instead of best-of-N as final aggregation (ablation) | VGS · competition math benchmarks | accuracy | Helped |
| use value-guided intermediate block selection instead of random block selection (ablation) | VGS · AIME-24 across inference budgets | accuracy | Helped |
| use a hybrid of random and value-guided search instead of pure VGS (ablation) | VGS · AIME-24 across inference budgets | accuracy | Hurt |
| use block size 4096 rather than finer-grained blocks (ablation) | beam search with width 2 · AIME-24 across inference budgets | accuracy | Helped |
| use beam width 2 rather than other tested widths, including BFS (ablation) | beam search with block size 4096 · AIME-24 across inference budgets | accuracy | Helped |
| add DVTS parallelism to a single search tree (ablation) | VGS with beam width 2 · AIME-24 across inference budgets | accuracy | Helped |
| use DeepSeek-1.5B-PPO instead of the base DeepSeek-1.5B policy | Pass@N · AIME-24 and HMMT-25 at Pass@32 and higher | accuracy: AIME-24 Pass@32: 74.1±3.2 vs. 73.3±3.4; HMMT-25 Pass@32: 37.8±5.1 vs. 36.9±4.4 | Hurt |
Surprising findings
- Qwen2.5-Math-PRM-7B and Math-Shepherd-PRM-7B give only marginal gains over MV, suggesting they are out-of-distribution for long DeepSeek CoTs.
- A 1.5B Bradley-Terry ORM trained on the same data is surprisingly effective as a search guidance model.
Limitations
- Value model is trained only on rollouts from DeepSeek-R1-Distill-Qwen-1.5B; gap to larger generators (14B) shrinks, suggesting distribution shift.
- Evaluation limited to competition math with binary outcome rewards; generalization to other domains untested.
- Small benchmark sizes (30 problems per competition) with high variance across seeds.
Experimental setup
Datasets: OpenR1-Math OpenR1-Cleaned OpenR1-VM
Benchmarks: AIME 2024 AIME 2025 HMMT Feb 2024 HMMT Feb 2025
Metrics: accuracy pass@N inference FLOPs
Baselines: majority voting weighted majority voting best-of-N Math-Shepherd-Mistral-7B-PRM Qwen2.5-Math-PRM-7B DeepSeek-BT-1.5B DeepSeek-R1 (671B) o3-mini-medium o1-mini-medium o4-mini-medium
Builds on: OpenR1-Math DeepSeek-R1-Distill Math-Shepherd PAV (process advantage verifiers) PPO
Closest papers by meaning
- DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains (ICLR 2026 · similarity 0.88)
- Language Models can Self-Improve at State-Value Estimation for Better Search (NeurIPS 2025 · similarity 0.87)
- Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization (ICLR 2026 · similarity 0.87)
- Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling (ICLR 2026 · similarity 0.86)
- ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Let's (not) just put things in Context: Test-time Training for Long-context LLMs (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.