Research map / Large language models / Retrieval and knowledge
Knowledge Fusion of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FuseLLM, which fuses several heterogeneous source LLMs (Llama-2, OpenLLaMA, MPT) into one target LLM by continually training it to match the source models' fused next-token distributions, using MinED token alignment and MinCE fusion.
Problem
Combining the capabilities of multiple pre-trained LLMs with different architectures and tokenizers is impractical via weight merging or ensembling, which require identical architectures or costly multi-model inference.
Approach
Externalize each source LLM's next-token distributions on a raw corpus, align them across tokenizers with minimum edit distance, fuse them with a min-cross-entropy rule, and train the target LLM to match the fused distribution alongside the CLM loss.
Key results
- On BBH (27 tasks), FuseLLM reaches 41.75 average EM vs 39.70 for Llama-2 (+5.16% relative) and 40.44 for Llama-2 CLM.
- On the 5-task commonsense benchmark, FuseLLM averages 64.56 vs 63.76 for Llama-2 (+1.25% relative).
- On MultiPL-E code generation, FuseLLM averages 15.56 pass@1 vs 14.63 for Llama-2 (+6.36%), but remains below OpenLLaMA (18.11).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fuse distributions from Llama-2, OpenLLaMA, and MPT and continually train the target | Llama-2 7B · BBH, Avg. 27 Tasks | exact match accuracy: 39.70 -> 41.75 | Helped |
| fuse distributions from Llama-2, OpenLLaMA, and MPT and continually train the target | Llama-2 7B · Common Sense, Avg. 5 Tasks | accuracy: 63.76 -> 64.56 | Helped |
| fuse distributions from Llama-2, OpenLLaMA, and MPT and continually train the target | Llama-2 7B · MultiPL-E, Avg. 10 Tasks | pass@1: 14.63 -> 15.56 | Helped |
| use three source models instead of two (ablation) | FuseLLM · BBH | exact match accuracy: 41.00 -> 41.75; 41.16 -> 41.75 | Helped |
| use three source models instead of two (ablation) | FuseLLM · Common Sense | accuracy: 64.50 -> 64.56; 64.51 -> 64.56 | No clear effect |
| use three source models instead of two (ablation) | FuseLLM · MultiPL-E | pass@1: 15.51 -> 15.56; 15.47 -> 15.56 | No clear effect |
| replace exact-match token alignment with MinED (ablation) | FuseLLM · BBH | exact match accuracy: 41.57 -> 41.75 | Helped |
| replace exact-match token alignment with MinED (ablation) | FuseLLM · MultiPL-E | pass@1: 15.49 -> 15.56 | Helped |
| replace exact-match token alignment with MinED (ablation) | FuseLLM · Common Sense | accuracy: 64.24 -> 64.56 | Helped |
| replace AvgCE weighted averaging with MinCE selection (ablation) | FuseLLM · BBH | exact match accuracy: 41.04 -> 41.75 | Helped |
| replace AvgCE weighted averaging with MinCE selection (ablation) | FuseLLM · MultiPL-E | pass@1: 15.39 -> 15.56 | Helped |
| replace AvgCE weighted averaging with MinCE selection (ablation) | FuseLLM · Common Sense | accuracy: 63.98 -> 64.56 | Helped |
Surprising findings
- Fusing into Llama-2 underperforms the strongest source model OpenLLaMA on code generation, attributed to limited code data in the continual-training corpus.
- LLM-Blender's rank-and-fuse variant degraded badly after fusion (BBH 24.48, ME 0.06), below its rank-only variant.
Limitations
- Fused target model still trails the strongest source (OpenLLaMA) on code generation.
- Relies on the training corpus relevance to downstream tasks; performance drops on tasks like Geometric Shapes and Word Sorting.
- Comparing against ensembles and merging only in a simulated setting with identical-architecture Pythia 1B models.
Experimental setup
Datasets: MiniPile The Pile RedPajama Evol-Instruct ShareGPT Open-Platypus
Benchmarks: Big-Bench Hard CommonSense (ARC, BoolQ, HellaSwag, OpenBookQA) MultiPL-E TriviaQA DROP LAMBADA IWSLT2017 SciBench Vicuna Benchmark
Metrics: exact match accuracy accuracy pass@1 BLEU bits per UTF-8 byte perplexity
Baselines: Llama-2 CLM OpenLLaMA MPT Llama-2 KD LLM-Blender Ensemble Weight Merging
Builds on: knowledge distillation LLM-Blender exact-match token alignment teacher annealing
Closest papers by meaning
- Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration (NeurIPS 2025 · similarity 0.91)
- LLM Augmented LLMs: Expanding Capabilities through Composition (ICLR 2024 · similarity 0.90)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.90)
- MindMerger: Efficiently Boosting LLM Reasoning in non-English Languages (NeurIPS 2024 · similarity 0.90)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.90)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.