Research map / Large language models / Language, safety and interpretability

Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a Gaussian-copula model of correlated LLM errors plus a greedy mutual-information selection rule for budgeted LLM ensemble selection, with a theoretical saturation floor, evaluated on MEDMCQA, MMLU, and IMDB.

Problem

Choosing which LLMs to query under a per-query budget when LLM errors are strongly correlated, so accuracy-based Top-k selection may be suboptimal and ensembles may saturate.

Approach

Models LLM errors with a Gaussian copula, proves Top-k optimal under independent errors, derives a decomposition of the marginal information gain (accuracy, redundancy, error correlation) that motivates a greedy mutual-information selection, and shows an error floor Phi(Phi^-1(1-alpha)/sqrt(rho)) under uniform correlation.

Key results

• On MEDMCQA, Greedy MI reaches its best error of 16.3% at k=5 versus 17.0% for Top-k selection.
• On MMLU, Greedy MI reaches about 0.144 error at k=4 versus 0.150 for Top-k at k=4.
• On IMDB, improvements from selection are modest given high average pairwise correlation (0.90) and accuracies of 91-96%.

Surprising findings

• Under independent errors Top-k is optimal, so the failure of accuracy-only selection is attributed entirely to correlation structure.
• Pure mRMR-style redundancy penalties pick weak models early and plateau around 17.3% error on MEDMCQA.

Limitations

• Restricted to binary true/false decisions, with multiple-choice converted to binary queries.
• MAP aggregation over 2^k patterns becomes noisy for large k with limited estimation data.
• Gains are modest on highly correlated tasks such as IMDB.

Experimental setup

Datasets: MEDMCQA MMLU IMDB movie reviews

Benchmarks: MEDMCQA MMLU IMDB

Metrics: test error probability

Baselines: Top-k by accuracy Term-1 only (relevance) Terms 1+2 (mRMR)

Builds on: mRMR Gaussian copula Brown et al. unified conditional mutual information feature selection self-consistency

Closest papers by meaning

• Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling (ICLR 2025 · similarity 0.88)
• Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration (NeurIPS 2024 · similarity 0.88)
• Rethinking LLM Ensembling from the Perspective of Mixture Models (ICML 2026 · similarity 0.88)
• How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.87)
• BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training (NeurIPS 2026 · similarity 0.87)
• Improving Pretraining Data Using Perplexity Correlations (ICLR 2025 · similarity 0.87)