Research map / Large language models / Language, safety and interpretability
Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies which factors govern LLM ensembling (performance gap, vocabulary size, response style), proposes a determine-then-ensemble model selection rule, and UniTE, a union top-k probability-level ensembling method that avoids full-vocabulary alignment.
Problem
Existing probability-level LLM ensembling methods ignore which models are compatible and align probabilities over the entire vocabulary at every step, which is costly and can fail with large vocabularies or mismatched response styles.
Approach
Ensemble only over the union of each model's top-k tokens, mapping tokens across tokenizers via the first sub-token when absent from a model's vocabulary, averaging normalized probabilities, and select base models by performance gap (<10%) and response length consistency.
Key results
- UniTE averages 66.54 across six benchmarks with OpenChat as primary model, vs 65.38 for OpenChat alone and 63.11 for DeePen.
- With LLaMA3 and Qwen2, UniTE reaches 81.14 average vs 78.81 for Qwen2 alone, while LLM-Blender and GaC fall below the best single model.
- UniTE manipulates about 14 tokens per step versus 32000-170336 for DeePen and GaC, with latency about 10 ms/token above single models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UniTE instead of LLM-Blender for the OpenChat + Mistral + DeepSeek ensemble | LLM ensemble · PIQA | accuracy: 83.28(-3.82) -> 87.50(+0.40) | Helped |
| Use UniTE instead of DeePen for the OpenChat + Mistral + DeepSeek ensemble | LLM ensemble · TriviaQA | accuracy: 67.24(+5.47) vs 65.80(+4.03) | Mixed |
| Use UniTE instead of LLM-Blender for the LLaMA3 + Qwen2 ensemble | LLM ensemble · MMLU | accuracy: 62.07 (-2.89) -> 69.78 (+4.82) | Helped |
| Add LLaMA3.1 to the LLaMA3 + Qwen2 UniTE ensemble | UniTE · PIQA | accuracy: 85.53 -> 84.98 | Mixed |
| Increase k from 5 to 10 (ablation) | UniTE with Mistral + OpenChat · TriviaQA | accuracy: 64.52 -> 65.80 | Helped |
| Increase k from 10 to 20 (ablation) | UniTE with Mistral + OpenChat · TriviaQA | accuracy: 65.80 -> 65.77 | Hurt |
| Increase k from 10 to 10000 (ablation) | UniTE with Mistral + OpenChat · TriviaQA | accuracy: 65.80 -> 65.72 | Hurt |
| Use greedy decoding instead of random sampling (ablation) | UniTE · PIQA and ARC-C | performance | Helped |
| Ensemble models with a substantial performance gap | LLM ensembling methods · GSM8K and other tasks | performance | Hurt |
| Ensemble models with different response styles on TriviaQA or NQ | LLM ensemble · TriviaQA and NQ | performance | Hurt |
| Increase vocabulary size disparity between compatible models | LLM ensembling methods · Ensembling benchmarks | performance | No clear effect |
Surprising findings
- Ensembling with a weaker model can outperform the stronger model when performance is comparable, e.g. DeePen on ARC-C.
- Differences in response style (e.g. LLaMA3 short answers vs Qwen2 long reasoning) can prevent successful ensembling even when performance matches.
Limitations
- Model selection relies on a heuristic (performance gap within 10%, response length within 2x) that is coarse and partly subjective.
- Results on several benchmarks are mixed; UniTE slightly underperforms DeePen on TriviaQA and GSM8K in some settings.
Experimental setup
Datasets: GSM8K PIQA ARC-C NQ MMLU TriviaQA CMMLU BBH
Benchmarks: MMLU ARC-C GSM8K PIQA TriviaQA NQ CMMLU BBH
Metrics: accuracy latency (ms/token) tokens manipulated per step
Baselines: LLM-Blender DeePen GaC
Builds on: DeePen GaC LLM-Blender
Closest papers by meaning
- Rethinking LLM Ensembling from the Perspective of Mixture Models (ICML 2026 · similarity 0.92)
- Ensembling Language Models with Sequential Monte Carlo (NeurIPS 2026 · similarity 0.91)
- Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration (NeurIPS 2024 · similarity 0.90)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.89)
- $\texttt{Model-GLUE}$: Democratized LLM Scaling for A Large Model Zoo in the Wild (NeurIPS 2024 · similarity 0.88)
- Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.