Research map / Large language models / AI and society

Are More LLM Calls All You Need? Towards the Scaling Properties of Compound AI Systems

NeurIPS 2024 · Large language models › AI and society · approach: Critiques of LLM claims and judging · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies how the number of LM calls affects majority-vote (Vote) and filter-then-vote (Filter-Vote) compound systems, showing non-monotonic scaling explained by query difficulty diversity and proposing an analytical scaling model to predict the optimal number of calls.

Problem

How the number of LM calls affects the performance of compound AI systems that aggregate multiple LM responses (e.g., majority voting, LM-based filtering), and how to choose that number.

Approach

Performance of Vote and Filter-Vote can be non-monotone in the number of LM calls because more calls help easy queries and hurt hard queries; a theory based on per-query difficulty indicators yields an analytical scaling model and a closed-form optimal number of calls for two-level difficulty tasks.

Key results

• Vote and Filter-Vote accuracy on AVERITEC rises then falls as LM calls grow from 2 to 1000, with the non-monotone trend explained by easy vs. hard query breakdown.
• The analytical scaling model fit on 2-100 calls predicts performance across MMLU PHYSICS, TruthfulQA, and GPQA with high correlation, more accurately for Vote than Filter-Vote.
• On synthetic bi-level difficulty data, the closed-form optimal number of LM calls exactly matches the empirically observed optimum across all 18 evaluated difficulty settings.

Measured effects reported in the paper

Surprising findings

• Performance of Vote and Filter-Vote often first increases then decreases (or U-shape) with more LM calls, so more calls is not always better.
• The best Filter-Vote performance is not necessarily better than Vote; on TruthfulQA Vote's best exceeds Filter-Vote's.

Limitations

• Focuses only on tasks with a small number of possible answers (e.g., multiple choice) that support majority voting; open-ended tasks not covered.
• Only two compound designs (Vote and Filter-Vote) and one LM (GPT-3.5-turbo) are studied.
• Does not account for the cost or latency of LM calls.

Experimental setup

Datasets: MMLU PHYSICS TruthfulQA GPQA AVERITEC

Benchmarks: MMLU PHYSICS TruthfulQA GPQA AVERITEC

Metrics: accuracy mean square error of predicted performance

Baselines: Vote Filter-Vote

Builds on: self-consistency AlphaCode 2 filtering Chain-of-thought

Closest papers by meaning

• Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.89)
• Representation Consistency for Accurate and Coherent LLM Answer Aggregation (NeurIPS 2025 · similarity 0.88)
• Provable Scaling Laws for the Test-Time Compute of Large Language Models (NeurIPS 2025 · similarity 0.88)
• A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.87)
• Efficient Evaluation of LLM Performance with Statistical Guarantees (NeurIPS 2026 · similarity 0.87)
• Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.87)