Research map / Large language models / Reasoning and chain of thought
Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeePEn, a training-free LLM ensembling method that fuses next-token probability distributions across heterogeneous-vocabulary LLMs by mapping them into a shared relative-representation space and inverting back via search.
Problem
Ensembling heterogeneous LLMs at the token level is blocked by vocabulary mismatch, and existing selection/fusion ensembles need trained reward or fusion models that generalize poorly to unseen distributions and base models.
Approach
Map each model's next-token distribution into a universal relative space (cosine similarities to shared anchor tokens, softmax-normalized), average the relative representations across models, and recover a main-model distribution by gradient-based search minimizing KL divergence to the aggregated representation.
Key results
- DeePEn-Adapt with top-4 models reaches +1.44 (TriviaQA) to +3.34 (ARC-C) over the best individual model.
- On the top-2 ensemble, LLM-Blender and MinED fail to improve consistently; MinED drops GSM8K by 35.40 points, while DeePEn-Avg improves on all six benchmarks.
- Combining DeePEn with Voting/MBR raises GSM8K from 63.15 to 65.25 over Voting alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse heterogeneous model distributions with DeePEn-Avg | Top-2 LLM ensembles · Six benchmarks | accuracy or exact match accuracy: +1.43(MMLU)~+2.72(PIQA) | Helped |
| Fuse heterogeneous model distributions with DeePEn-Avg | Top-4 LLM ensembles · Six benchmarks | accuracy or exact match accuracy: +1.00(PIQA)~+2.89(ARC-C) | Mixed |
| Weight models by development-set performance instead of averaging uniformly | DeePEn top-4 ensemble · ARC-C | accuracy: 78.70 -> 79.15 | Helped |
| Add Voting to DeePEn, using MBR on TriviaQA and NQ | Top-4 DeePEn ensemble · GSM8K | exact match accuracy: 63.15→65.25 | Helped |
| Add the third and fourth ranked models to the top-2 ensemble | DeePEn-Avg · Six benchmarks | accuracy or exact match accuracy | Mixed |
| Align tokens by minimum edit distance | MinED · GSM8K, top-2 ensemble | exact match accuracy: -35.40 | Hurt |
| Use full common-token anchors rather than fewer anchors (ablation) | DeePEn · MMLU and ARC-C | accuracy | Helped |
| Normalize relative representation rows with softmax (ablation) | DeePEn · Ensemble with outlier words | accuracy | Helped |
| Use different relative ensemble learning rates (ablation) | DeePEn top-2 ensemble · MMLU | improvement over best individual model: +2.42; +1.57; +1.77; +1.96; +1.31; +1.31 | Mixed |
| Use different relative ensemble learning rates (ablation) | DeePEn top-2 ensemble · TriviaQA | improvement over best individual model: +1.31; +2.05; +1.63; +1.94; +1.82; +1.26 | Mixed |
| Increase the number of inverse-transformation learning steps (ablation) | DeePEn · Downstream tasks | accuracy | Mixed |
| Choose different main models for inverse transformation (ablation) | DeePEn · MMLU-Dev and ARC-C-Dev | accuracy: 63.63 to 64.46; 77.73 to 77.55 | Mixed |
Surprising findings
- MinED causes a 35.40-point drop on GSM8K in the top-2 ensemble, attributed to token misalignment producing incomplete words.
- Under-fitting in the inverse search lets DeePEn slightly outperform vanilla prediction averaging on MMLU for same-vocabulary models.
Limitations
- Performance can drop when adding weaker models due to interference; fixed model-level weights only.
- Inverse search underfits, so the main model's choice affects results and the method is not invariant to it.
- Adds inference latency from T-step search.
Experimental setup
Datasets: MMLU ARC-C GSM8K PIQA TriviaQA NQ Flores-200 MixInstruct
Benchmarks: MMLU ARC-C GSM8K PIQA TriviaQA NQ Flores-200
Metrics: accuracy exact match
Baselines: LLM-Blender MinED Voting MBR EVA
Builds on: relative representation LLM-Blender MinED MBR
Closest papers by meaning
- Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling (ICLR 2025 · similarity 0.90)
- Ensembling Language Models with Sequential Monte Carlo (NeurIPS 2026 · similarity 0.89)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.89)
- Rethinking LLM Ensembling from the Perspective of Mixture Models (ICML 2026 · similarity 0.89)
- Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection (NeurIPS 2026 · similarity 0.88)
- Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.