Research map / Large language models / Reasoning and chain of thought
Rollout Roulette: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes particle filtering over an LLM-PRM state space model for inference-time scaling, reporting 4-16x better scaling than deterministic search (beam search, DVTS) on math and general reasoning tasks.
Problem
Deterministic inference-time scaling methods (beam search, MCTS, DVTS) guided by imperfect process reward models prune promising trajectories early, leading to suboptimal answers.
Approach
Formulate inference-time scaling as posterior inference in a state space model (LLM as transition, PRM as Bernoulli emission) and apply particle filtering with softmax resampling over PRM scores, maintaining a diverse particle population instead of greedy pruning. Also extends to particle Gibbs and parallel tempering variants.
Key results
- Qwen2.5-Math-7B-Instruct with particle filtering at budget 32 reaches 87.7% on MATH500 and 10/30 on AIME 2024, matching or exceeding o1-preview on MATH500.
- Qwen2.5-1.5B-Instruct with particle filtering (budget 32) reaches 79.3% on MATH500 versus 76.2% for GPT-4o.
- Particle filtering scales 4-16x faster than DVTS in budget needed to reach the same accuracy on math SLMs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use particle filtering instead of the strongest listed inference-time baseline | Qwen2.5-Math-7B-Instruct · MATH500 | accuracy: 86.9 -> 87.7 | Helped |
| Use particle filtering instead of the strongest listed inference-time baseline | Qwen2.5-Math-7B-Instruct · AIME 2024 | accuracy: 7/30 -> 10/30 | Helped |
| Use particle filtering instead of the strongest listed inference-time baseline | Llama-3.1-8B-Instruct · FinanceBench | accuracy: 68.67 -> 70.33 | Helped |
| Use particle filtering instead of the strongest listed inference-time baseline | Llama-3.1-8B-Instruct · NumGLUE Task 2 (Chemistry) | accuracy: 80.92 -> 84.22 | Helped |
| Use Qwen2.5-Math-PRM-7B rather than the other tested reward models (ablation) | Particle Filtering · MATH500 (100-question subset) | accuracy | Helped |
| Use model-based reward aggregation rather than step-level aggregation (ablation) | Particle Filtering · MATH500 | accuracy | Helped |
| Use generation temperatures in the 0.4--1.0 range (ablation) | Particle Filtering · MATH questions (100-question subset) | accuracy | No clear effect |
| Allocate compute to multiple Particle Gibbs iterations rather than an equivalently budgeted single PF run (ablation) | Particle Gibbs · MATH-500 (100-question subset) | accuracy | No clear effect |
| Increase the number of parallel chains (ablation) | Parallel tempering · MATH500 (100-question subset) | accuracy | Helped |
Surprising findings
- Qwen2.5-Math-PRM-7B, trained primarily for math, performs surprisingly well as a reward model on FinanceBench and NumGLUE chemistry.
- Model-based aggregation (PRM used as an ORM on partial answers) scales best, outperforming product, min, and last-step aggregation.
Limitations
- Hosting and running a reward model adds high latency and resource cost.
- Smaller models often need prompt engineering to adhere to output format.
- Budget hyperparameters are problem-dependent and may need tuning across domains.
Experimental setup
Datasets: MATH500 AIME 2024 NumGLUE Task 2 (Chemistry) FinanceBench
Benchmarks: MATH500 AIME 2024 NumGLUE Task 2 (Chemistry) FinanceBench
Metrics: accuracy
Baselines: Greedy Self Consistency BoN WBoN Beam Search DVTS TSMC
Builds on: beam search diverse verifier tree search (DVTS) particle Gibbs self-consistency best-of-N
Closest papers by meaning
- Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scaling (ICML 2026 · similarity 0.91)
- Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling (NeurIPS 2025 · similarity 0.91)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.91)
- OptScale: Probabilistic Optimality for Inference-time Scaling (AAAI 2026 · similarity 0.90)
- Entropy-Gated Latent Recursion (NeurIPS 2026 · similarity 0.90)
- Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.