Research map / Large language models / Reasoning and chain of thought
Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlo
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes verifying LLM math solutions with Twisted Sequential Monte Carlo, using a learned value function as twist so resampling focuses on promising partial solutions, without step-level human labels.
Problem
Verification-based math reasoning with LLMs is sample-inefficient (many samples needed for one correct solution) and training strong process reward models requires costly step-wise supervision.
Approach
Views weighted majority voting as importance sampling and replaces it with TSMC: at each reasoning step, partial solutions are reweighted by the square root of a learned value function and resampled; the value function is trained with contrastive twist learning on outcome-labeled samples only.
Key results
- On MATH500 with DeepSeek-7B, TSMC + WMV reaches 60.8% versus 55.0% for WMV with ORM and 52.8% for MV.
- On GSM8K with Llemma-7B, TSMC + WMV reaches 80.4% versus 79.2% for the best WMV baseline (SHEPHERD PRM).
- On NumGLUE with CodeLlama-7B at N=5, TSMC + WMV reaches 99.0% versus 97.0% for WMV with ORM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TSMC + WMV instead of the strongest reported baseline | Llemma-7B generator · GSM8K | problem solving rate (%): 79.2 -> 80.4 | Helped |
| Use TSMC + WMV instead of the strongest reported baseline | Llemma-7B generator · MATH500 | problem solving rate (%): 43.6 -> 46.4 | Helped |
| Use TSMC + WMV instead of the strongest reported baseline | DeepSeek-7B generator · GSM8K | problem solving rate (%): 89.5 -> 91.7 | Helped |
| Use TSMC + WMV instead of the strongest reported baseline | DeepSeek-7B generator · MATH500 | problem solving rate (%): 55.2 -> 60.8 | Helped |
| Use TSMC-generated solutions with majority voting instead of vanilla majority voting | Llemma-7B generator · GSM8K | problem solving rate (%): 72.5 -> 78.1 | Helped |
| Use TSMC-generated solutions with majority voting instead of vanilla majority voting | DeepSeek-7B generator · MATH500 | problem solving rate (%): 52.8 -> 55.6 | Helped |
| Use step-correctness predictions as the incremental importance weights (ablation) | TSMC with Llemma-7B · MATH500 | problem solving rate | No clear effect |
| Replace the estimated V^p twist with PRM (SHEPHERD) value estimates (ablation) | TSMC with Llemma-7B · MATH500 | problem solving rate | Hurt |
| Use V^p(x_{1:t}) instead of sqrt(V^p(x_{1:t})) as the twist (ablation) | TSMC with Llemma-7B · MATH500 | problem solving rate | Hurt |
| Reduce the TSMC batch size to M=1 (ablation) | TSMC · MATH500 | problem solving rate | Hurt |
| Use process-reward product aggregation instead of last-step aggregation for estimated value rewards (ablation) | PRM (SHEPHERD) and Value (TSMC) · math benchmarks | problem solving rate | Hurt |
| Use process-reward minimum aggregation instead of product aggregation for estimated value rewards (ablation) | PRM (SHEPHERD) and Value (TSMC) · math benchmarks | problem solving rate | Helped |
Surprising findings
- Using the product aggregation of step-level estimated values is consistently poor, as values in [0,1] bias scores toward solutions with fewer steps.
- Using the step-correctness PRM as the intermediate twist does not improve sampling efficiency because it ignores the generator's likelihood.
Limitations
- Variable step lengths break parallelism, adding inference latency.
- Batch-based resampling makes TSMC a batch decoding method whose performance depends on batch size; very small batches are not robust.
- Gains over strong baselines are modest on some benchmarks (e.g., GSM8K with Llemma-7B: 80.4 vs 79.2), and results are on 7B models only.
Experimental setup
Datasets: GSM8K MATH PRM800K MATH-SHEPHERD MultiArith NumGLUE
Benchmarks: GSM8K MATH500 MultiArith NumGLUE MATH validation
Metrics: problem solving rate (%)
Baselines: Greedy decoding majority voting WMV with ORM WMV with PRM (PRM800K) WMV with PRM (SHEPHERD) best-of-K step-level beam search
Builds on: Twisted Sequential Monte Carlo Contrastive Twist Learning Math-Shepherd PRM800K
Closest papers by meaning
- Improving Value-based Process Verifier via Low-Cost Variance Reduction (AAAI 2026 · similarity 0.89)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.88)
- MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.88)
- MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data (ICLR 2024 · similarity 0.88)
- MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy (AAAI 2026 · similarity 0.88)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.