Research map / Large language models / Reasoning and chain of thought
Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically studies inference scaling laws and compute-optimal inference for math reasoning, and proposes REBASE, a reward-balanced tree search that beats sampling and MCTS at matched FLOPs, letting smaller models match larger ones.
Problem
How to trade off model size against inference-time compute (sampling, voting, best-of-n, tree search) so that LLM problem-solving accuracy is maximized under a fixed FLOPs budget.
Approach
Formulates compute-optimal inference as jointly choosing model size and inference strategy under a FLOPs budget; proposes REBASE, a tree search that allocates expansion width per node via softmax-normalized process reward scores, avoiding MCTS rollouts, and pairs it with weighted voting.
Key results
- REBASE with Mistral-7B and weighted voting reaches 45.0% on MATH500 at 1.36e14 FLOPs versus 42.8% for sampling at 8.70e14 FLOPs (256 samples).
- Llemma-7B with REBASE reaches 46.8% on MATH500 at 1.48e14 FLOPs, exceeding sampling with Llemma-34B-scale compute (Llemma-34B sampling 46.7% at 12.1e14 FLOPs for 64 samples).
- Fitted compute-optimal relation log10(C)=1.19 log10(N)+2.03 relates inference FLOPs to optimal model size for Pythia on GSM8K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace sampling with REBASE tree search (weighted voting) | Llemma-7B and Llemma-34B, Mistral-7B · MATH500 | accuracy: Llemma-7B 46.8 at 1.48e14 FLOPs vs sampling 45.5 at 10.0e14 | Helped |
| Use smaller Llemma-7B with REBASE instead of Llemma-34B with sampling | Llemma-7B · MATH500 | accuracy: Llemma-7B REBASE 46.8 vs Llemma-34B sampling 46.7 at lower compute | Helped |
| Replace sampling with REBASE tree search | Llama3-Instruct-8B · GSM8K / MATH500 | accuracy: MATH: 43.7 (REBASE+WV) vs 39.5 (Sampling+WV) at 32 samples | Helped |
| Replace MCTS with sampling-based methods at matched compute (ablation) | MCTS tree search · MATH500 and GSM8K | accuracy vs compute | Hurt |
| Use REBASE tree search instead of sampling-based weighted voting | Mistral-7B · MATH500 | accuracy: 45.0 vs 42.8 with about 6x fewer FLOPs | Helped |
| Replace sampling with REBASE on MBPP code generation | Llama3-8B-Instruct · MBPP | pass rate: 81.4 vs 79 at 64 samples | Helped |
| Gain from REBASE over sampling on hard problems (ablation) | REBASE vs sampling, MATH levels 3-5 · MATH hard vs easy | accuracy | Helped |
| Scale inference compute with weighted majority voting on a fixed model | Pythia models · GSM8K | error rate | Helped |
| Use weighted majority voting instead of majority voting for sampling (ablation) | sampling-based candidates · MATH500 and GSM8K | accuracy | Helped |
| Use REBASE rather than sampling on Llemma-34B at 8 samples | Llemma-34B · MATH500 | accuracy: REBASE+MV 43.9 vs Sampling+MV 39.9 | Helped |
| Replace sampling with REBASE on GSM8K for Llemma-34B (ablation) | Llemma-34B · GSM8K | accuracy: REBASE+WV 86.9 vs Sampling+BoN 86.7 (8 samples); REBASE+MV 86.1 vs sampling+MV 84.3 | Mixed |
Surprising findings
- MCTS underperforms simple sampling-based methods at each compute budget when paired with weighted voting, due to many unfinished solutions and costly rollouts.
- Theory shows sampling-based majority voting saturates to a limit set by the model's output distribution, so more samples alone cannot reach perfect accuracy.
Limitations
- Reward model is Llemma-34B fine-tuned on Math-Shepherd, so results depend on a strong PRM
- Experiments restricted mainly to math reasoning, with limited code evidence (MBPP in appendix)
- Theoretical analysis does not formally characterize tree search behavior
Experimental setup
Datasets: MetaMath GSM8K MATH Math-Shepherd MBPP
Benchmarks: MATH500 GSM8K MBPP
Metrics: accuracy error rate FLOPs
Baselines: Greedy search Sampling + majority voting Sampling + best-of-n Sampling + weighted voting MCTS
Builds on: Monte Carlo Tree Search self-consistency Math-Shepherd MetaMath Chinchilla scaling laws
Closest papers by meaning
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.93)
- Provable Scaling Laws for the Test-Time Compute of Large Language Models (NeurIPS 2025 · similarity 0.93)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.92)
- OptScale: Probabilistic Optimality for Inference-time Scaling (AAAI 2026 · similarity 0.92)
- Bag of Tricks for Inference-time Computation of LLM Reasoning (NeurIPS 2025 · similarity 0.92)
- Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.