Research map / Large language models / Reasoning and chain of thought
Rethinking LLM Ensembling from the Perspective of Mixture Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reframes LLM ensembling as a mixture model: at each step one model is sampled by ensemble weights and only it runs a forward pass, giving the same token distribution as conventional ensembling at 1.78-2.68x higher speed.
Problem
Conventional LLM ensembling requires a forward pass from every model at each decoding step to compute the averaged output distribution, which is expensive; the paper asks whether all models truly need to be invoked.
Approach
Treat the averaged ensemble distribution as a mixture: sample a model index from the ensemble weights at each step and sample the next token only from that model's distribution. This is provably equivalent in token distribution to sampling from the full ensemble, and it links LLM ensembling to token-level routing (ensembling is the simplest random router). Also introduces lazy KV-cache synchroniza
Key results
- ME matches conventional ensemble accuracy across two-, three-model and heterogeneous ensembles (e.g., Qwen-3B + Qwen-Math-1.5B GSM8K 82.97 vs CE 83.14 at k=5).
- ME achieves 1.78x-2.68x speedup over sequential conventional ensembling on H100 across four model configurations (e.g., 46.17 vs 17.21 tokens/s for the heterogeneous three-model ensemble).
- Adding more models does not always help: the 3-model Openchat+Nous-Hermes+OpenHermes ensemble does not beat the 2-model ensemble on most datasets.
Surprising findings
- Adding more models does not always improve ensemble accuracy; the 3-model ensemble underperforms the 2-model one on most tasks.
- Parallel conventional ensembling gives only a small speedup over sequential ensembling and is slower than sequential on RTX 3090 due to inter-GPU communication.
Limitations
- Equivalence holds only for probabilistic sampling; under greedy decoding the mixture view fails and every model still needs a forward pass.
- Gains in accuracy over single models are modest; ME offers little performance improvement beyond conventional ensembling and provides no routing intelligence.
- Ensemble weights are chosen by grid search on test-related evaluation with step 0.1, and results are averaged over only five runs.
Experimental setup
Datasets: GSM8K MMLU BBH ARC
Benchmarks: GSM8K MMLU BBH ARC
Metrics: accuracy tokens per second
Baselines: Conventional Ensemble (sequential) Conventional Ensemble (parallel) Single Model
Builds on: UniTe speculative decoding top-k ensembling
Closest papers by meaning
- Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling (ICLR 2025 · similarity 0.92)
- Ensembling Language Models with Sequential Monte Carlo (NeurIPS 2026 · similarity 0.91)
- SpecEM: Training-Free LLM Ensembling via Iterative Drafting, Verification, and Online Feedback (NeurIPS 2025 · similarity 0.89)
- Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration (NeurIPS 2024 · similarity 0.89)
- $\texttt{Model-GLUE}$: Democratized LLM Scaling for A Large Model Zoo in the Wild (NeurIPS 2024 · similarity 0.88)
- Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.