Research map / Large language models / Reasoning and chain of thought
Token-Supervised Value Models for Enhancing Mathematical Problem-Solving Capabilities of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes token-supervised value models (TVMs), verifiers trained with per-token probabilities of reaching the correct answer, which improve tree-search inference (beam search, REBASE) on GSM8K and MATH over ORMs and PRMs.
Problem
Existing verifiers (ORMs and PRMs) were designed for Best-of-N search and are suboptimal for tree search at test time: ORMs give only implicit partial-solution signals and PRMs have high false negative rates, prematurely pruning promising steps in math reasoning.
Approach
Train the verifier with token-level labels equal to the empirical probability that a partial solution (prefix up to that token) leads to the correct final answer, estimated from sampled rollouts by counting. This is framed as value estimation (expected cumulative reward with gamma=1), so tree search can be value-guided, and it reduces false negatives relative to step-level PRM labels.
Key results
- On GSM8K, TVM with REBASE reaches 89.21% for Mistral 7B MetaMath and 90.35% with Step-by-Step Beam Search for Llama 3 8B MetaMath, outperforming ORM and PRM under the same search.
- On MATH, TVM with Step-by-Step Beam Search reaches 39.33% (Mistral 7B MetaMath) and 45.00% (Llama 3 8B MetaMath), above best Best-of-N results (37.40% and 44.40%).
- Tree search with TVM uses roughly 6x fewer FLOPs and 2x less execution time than Best-of-N search with N=256.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the ORM with a TVM for Best-of-N search | Mistral 7B on GSM8K · GSM8K | accuracy: 85.52 -> 88.17 | Helped |
| replace the ORM with a TVM for Step-by-Step Beam Search | Mistral 7B on GSM8K · GSM8K | accuracy: 86.73 -> 87.69 | Helped |
| replace the ORM with a TVM for REBASE | Mistral 7B on GSM8K · GSM8K | accuracy: 86.81 -> 87.97 | Helped |
| replace the PRM with a TVM for Best-of-N search | Llama 3 8B MetaMath on MATH · MATH | accuracy: 44.40 -> 43.40 | Hurt |
| replace the PRM with a TVM for Step-by-Step Beam Search | Llama 3 8B MetaMath on MATH · MATH | accuracy: 42.20 -> 45.00 | Helped |
| replace the ORM with a TVM for token-level value estimation | Mistral 7B on GSM8K · GSM8K test | RMSE: 0.2813 -> 0.2575 | Helped |
| increase beam search settings from K=20, b=5 to K=40, b=10 (ablation) | TVM + Step-by-Step Beam Search, Mistral 7B · GSM8K | accuracy: 86.05 -> 87.69 | Helped |
| increase beam search settings from K=40, b=10 to K=100, b=25 (ablation) | TVM + Step-by-Step Beam Search, Mistral 7B · GSM8K | accuracy: 87.69 -> 87.92 | Helped |
Surprising findings
- Math-Shepherd PRM underperforms ORM under Step-by-Step Beam Search on GSM8K for Mistral 7B MetaMath (86.66 vs 87.79), suggesting PRMs can harm tree search.
- TVM also improves over ORM and PRM under Best-of-N search on GSM8K, not only under tree search.
Limitations
- Gains on GSM8K are small (often under 1 point) and the MATH Best-of-N result for Llama 3 8B MetaMath favors PRM by 1.0 point.
- Training requires sampling many rollouts per training problem (N_tr=100 or 25), and the token-level labels are only informative when reasoning paths overlap.
- Evaluated only on math word problems and LLMs under 10B parameters.
Experimental setup
Datasets: GSM8K MATH MetaMathQA
Benchmarks: GSM8K MATH
Metrics: accuracy RMSE of value estimates precision-recall false negative error FLOPs execution time
Baselines: Self-Consistency ORM PRM (Math-Shepherd) Best-of-N search REBASE Step-by-Step Beam Search
Builds on: Math-Shepherd REBASE Step-by-Step Beam Search MetaMath vLLM
Closest papers by meaning
- Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning (AAAI 2024 · similarity 0.89)
- Generative Verifiers: Reward Modeling as Next-Token Prediction (ICLR 2025 · similarity 0.89)
- Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.89)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.88)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.88)
- MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.