Research map / Large language models / Reasoning and chain of thought

Improving Value-based Process Verifier via Low-Cost Variance Reduction

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ComMCS, an unbiased variance-reduction method that linearly combines Monte Carlo value estimates of the current and next reasoning step to train value-based process verifiers without extra LLM inference cost.

Problem

Value-based process verifiers for LLM math reasoning are trained on Monte Carlo value estimates whose error is dominated by variance due to the limited number of rollouts affordable under LLM inference cost.

Approach

Shows the MC estimator is an MVUE under a binomial model, then builds a lower-variance unbiased target by linearly combining current-step and next-step MC estimates (TD-inspired), with coefficients chosen by a heuristic variance comparison under a Gaussian/categorical approximation of the one-step value distribution, trained with cross-entropy.

Key results

• On MATH-500 with Deepseek-math-7b-instruct, Best-of-32 accuracy rises from 23.6 (V_CE w/o ComMCS) to 25.8 with ComMCS, a 2.2-point gain.
• On GSM8K with Qwen2.5-Math-7B-Instruct at Best-of-128, V_CE with ComMCS reaches 91.1 vs 90.4 for V_MSE.
• Beam search on MATH-500 with Deepseek-math-7b-instruct improves from 46.4 to 47.2 with ComMCS.

Surprising findings

• Value-distribution (CE) verifiers do not uniformly beat MSE/BCE baselines; V_MSE and V_BCE outperform V_CE without ComMCS on MATH-500 with Deepseek-math-7b-instruct.
• On Qwen2.5-Math-7B-Instruct MATH-500, the ComMCS and non-ComMCS CE variants perform similarly.

Limitations

• Relies on a Gaussian distribution hypothesis for one-step value distributions, which may not hold across tasks.
• Gains are modest and on Qwen2.5-Math-7B MATH-500 the improvement is negligible.
• Only evaluated on math reasoning (MATH-500, GSM8K); not tested on code.

Experimental setup

Datasets: MATH MetaMath GSM8K MATH-500

Benchmarks: MATH-500 GSM8K

Metrics: best-of-N accuracy beam search accuracy oracle coverage

Baselines: V_MSE regression verifier V_BCE return-distribution verifier V_CE without ComMCS

Builds on: Math-Shepherd temporal difference learning distributional reinforcement learning (C51) MetaMath

Closest papers by meaning

• AlphaMath Almost Zero: Process Supervision without Process (NeurIPS 2024 · similarity 0.89)
• Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlo (ICLR 2025 · similarity 0.89)
• Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data (AAAI 2024 · similarity 0.87)
• Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals (ICML 2026 · similarity 0.87)
• Token-Supervised Value Models for Enhancing Mathematical Problem-Solving Capabilities of Large Language Models (ICLR 2025 · similarity 0.87)
• From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.87)