Research map / Large language models / Reasoning and chain of thought

The Limits of Inference Scaling Through Resampling

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that resampling against imperfect unit-test verifiers caps accuracy because false positives cannot be filtered out, so weaker models cannot match stronger single-sample accuracy on HumanEval+ and MBPP+, and that optimal sample counts are finite and low when false positives carry a cost.

Problem

Inference scaling by resampling against imperfect verifiers (unit tests with limited coverage) may not let weaker models match stronger ones, because verifier false positives cannot be reduced by more sampling.

Approach

Formalizes that resampling cannot reduce verifier false-positive probability, deriving an accuracy ceiling set by the single-sample accuracy of stronger models; empirically links weaker models' higher false-positive rates to this ceiling and adds a cost for false positives to find finite optimal sample counts.

Key results

• On HumanEval+ and MBPP+, weaker models show higher false-positive rates, with conditional accuracy after passing base tests scaling inversely with single-sample capability.
• With a false-positive cost-benefit ratio of 4, the optimal number of samples is K <= 5 for all four evaluated models on HumanEval+.
• False-positive solutions have lower code-quality compliance (e.g., camelCase, line length, comments) than true positives across all models tested.

Surprising findings

• Optimal sample count can be zero when the false-positive cost exceeds the true-positive benefit, making resampling useless for weak models.
• Some models (e.g., Llama-3.1 70B) show a sharply rising false-positive rate with K, while others rise gradually, with no intuitive explanation found.

Limitations

• Only coding tasks (HumanEval+, MBPP+) studied; other domains may differ.
• Uses human-written unit tests as verifiers; LLM-generated tests not evaluated.
• Does not identify which cost-benefit ratio applies in real deployments; excluded tasks with oracle issues.

Experimental setup

Datasets: HumanEval+ MBPP+ HumanEval MBPP

Benchmarks: HumanEval+ MBPP+

Metrics: false positive rate conditional accuracy given passing base unit tests reward under cost-benefit ratio code readability compliance rate

Baselines: GPT-4o

Builds on: HumanEval+ MBPP+ inference scaling laws (Large Language Monkeys) EvalPlus

Closest papers by meaning

• Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.87)
• Tail Competition Explains Scaling in Best-of-N Sampling for Verifiable Problems (NeurIPS 2026 · similarity 0.87)
• Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.87)
• Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification (ICML 2025 · similarity 0.87)
• OptScale: Probabilistic Optimality for Inference-time Scaling (AAAI 2026 · similarity 0.86)
• A Simple Model of Inference Scaling Laws (ICML 2025 · similarity 0.86)