Research map / Large language models / Reasoning and chain of thought

Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Compute

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ModelSwitch, a training-free multi-LLM repeated-sampling strategy that splits a sample budget across several LLMs, stopping early when one model's answers are fully consistent, to improve accuracy and cut sampling cost.

Problem

Repeated sampling with voting needs many samples to reach high accuracy, which is costly and slow; the paper asks whether sample efficiency can be improved by generation-side diversity across multiple LLMs.

Approach

Use answer consistency (low entropy across samples) as a stopping signal: sample each LLM an equal share of the budget in sequence, switch to the next model when the current one is not unanimous, and aggregate all answers with internal (entropy-based) and external (prior performance) weighted voting.

Key results

• ModelSwitch reaches 81% on MATH with about 35 samples, beating Gemini 1.5 Flash self-consistency (79.8% at 512 samples) at roughly 14x lower sampling.
• On MMLU-Pro, ModelSwitch achieves 63.2% accuracy, 10.2 points above the best single LLM and above MOA (52.6%), MAD (47.6%) and AgentVerse (44.8%) under a 15-sample budget.
• Across six datasets at budget 16, ModelSwitch saves 34% of samples on average, up to 43% on GSM8K.

Surprising findings

• Two-model switching with a weaker second model can beat the stronger single model with the same budget, even when the weaker model is much less accurate on MMLU-Pro (gap above 10 points).
• Adding more than two models gave plateaus or declines (e.g., DATE dropped from 78.6% to 76.4% going from 2 to 6 models), so few comparable models suffice.

Limitations

• Efficiency gains are limited on hard problems, e.g., AIME24 saved only about 9% of samples since models rarely reach fully consistent answers.
• Theoretical analysis is restricted to two models and a per-query expectation; the multi-model bound relies on assumed per-model probabilities of consistency.
• Experiments rely mostly on closed-source API models, so reproducibility depends on API access.

Experimental setup

Datasets: GSM8K MATH MathBench MGSM DATE MMLU-Pro AIME24

Benchmarks: GSM8K MATH MathBench MGSM DATE MMLU-Pro AIME24

Metrics: accuracy average sampling count API cost

Baselines: self-consistency MAD MAD (MLD) ChatEval AgentVerse MOA GPT-4o Gemini 1.5 Pro Llama-3.1-70B-Instruct

Builds on: self-consistency chain-of-thought multi-agent debate mixture of agents

Closest papers by meaning

• GuidedSampling: Steering LLMs Towards Diverse Candidate Solutions at Inference-Time (ICLR 2026 · similarity 0.89)
• Evaluating Test-Time Scaling of General LLM Agents (NeurIPS 2026 · similarity 0.89)
• A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.88)
• Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification (ICML 2025 · similarity 0.88)
• Scaling Test-Time Compute Without Verification or RL is Suboptimal (ICML 2025 · similarity 0.87)
• Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time Scaling (ICLR 2026 · similarity 0.87)