Research map / Large language models / Reasoning and chain of thought

On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proves that parallel best-of-n is suboptimal under a mixture-of-reference-policies pretraining model and proposes reward-filtered sequential BoN (RF-SeqBoN), which feeds only high-reward generations back into the context, with gains on math/science reasoning benchmarks.

Problem

Test-time compute with reward models: the fundamental sample-complexity limits of parallel best-of-n are unclear, and sequential revision lacks theoretical understanding; the paper asks how to best allocate a fixed test-time budget.

Approach

Models the LLM as a mixture of reference policies from pretraining, shows a sequential lower bound that is strictly below the parallel BoN requirement, and proposes RF-SeqBoN that appends to the prompt only generations whose reward exceeds a threshold gamma, concentrating the history on high-reward candidates.

Key results

• RF-SeqBoN consistently outperforms vanilla BoN and PureSeq across the reported benchmarks with Qwen3-4B-Instruct and Qwen3-0.6B-Thinking (mean over five runs).
• On MATH500 and AIME'24 the advantage of RF-SeqBoN persists or grows with larger budgets; on GPQA-Diamond, AMC'23 and AIME'25 baselines catch up at high budgets.
• RF-SeqBoN beats Top-k refinement at all budgets, with the largest margin at small N.

Surprising findings

• Pure sequential revision performs slightly worse than BoN on easier MATH levels 1-3 while gaining on levels 4-5.
• With an outcome reward model, the adapted method shows a minor performance drop at large budgets, attributed to possible reward over-optimization.

Limitations

• Theory requires epsilon small relative to 1 - gamma* and comparator supported on high-reward actions; general epsilon left open.
• Assumption on reward-conditioned reference policy margin may fail, risking reward hacking in the feedback loop.
• Experiments use a single PRM and limited model sizes; ORM ablation shows degradation at large budgets.

Experimental setup

Benchmarks: MATH500 GPQA-Diamond AMC'23 AIME'24 AIME'25

Metrics: exact-match accuracy multiple-choice accuracy accuracy vs. generation budget N

Baselines: BoN PureSeq (pure sequential BoN) Top-k refinement

Builds on: best-of-n sampling E_M-divergence coverage analysis rewind-and-repeat selection inference

Closest papers by meaning

• Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment (ICML 2025 · similarity 0.92)
• Compute Aligned Training: Optimizing for Test Time Inference (NeurIPS 2026 · similarity 0.90)
• Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models (ICLR 2025 · similarity 0.90)
• Optimizing Test-Time Compute via Meta Reinforcement Finetuning (ICML 2025 · similarity 0.90)
• Tail Competition Explains Scaling in Best-of-N Sampling for Verifiable Problems (NeurIPS 2026 · similarity 0.90)
• Inference-Time Reward Hacking in Large Language Models (NeurIPS 2025 · similarity 0.90)