Research map / Label-efficient and robust learning / Test-time adaptation
Fusing Models with Complementary Expertise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fusion of Experts (FoE), training a small fuser on concatenated expert outputs (or LLM embeddings) to route inputs to complementary domain experts, plus FrugalFoE, a kNN-based sequential expert-selection method framed as a shortest-path problem to reduce expert calls.
Problem
Test data drawn from a mixture of domains where no single expert performs well; combining complementary pre-trained experts (classifiers, LLMs, summarizers, metric linear models) while limiting costly expert evaluations at test time.
Approach
Train a fuser via supervised learning on validation data that maps concatenated expert outputs (or averaged last-layer LLM embeddings) to the index of the correct expert, and make expert querying frugal by a kNN-estimated expected-loss stopping rule cast as an A*-like graph shortest path.
Key results
- On CIFAR-100 superclass classification with 20 experts, FoE reaches 82.13% final accuracy vs 87.63% oracle and 76.64% ensemble; FrugalFoE matches full-expert accuracy using 37.5% of experts.
- On sentiment analysis across four experts, FoE achieves 91.88% average accuracy with 99.1% expert selection accuracy, close to oracle.
- On summarization across six Pegasus experts, FoE attains 23.72 average ROUGE-2 versus 23.7 oracle-level performance, with single-expert features nearly matching full FoE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse complementary expert outputs with FoE | ResNet-18 experts · CIFAR-100 super-class classification test | Final accuracy (%): 76.64 -> 82.13 | Helped |
| Fuse complementary expert outputs with FoE | Fine-tuned sentiment models · Sentiment analysis average | Accuracy (%): 79.44% -> 91.88% | Helped |
| Use only CNN DM expert features (ablation) | Summarization FoE · Summarization average | ROUGE-2: 23.7171 -> 23.7180 | No clear effect |
| Use only Multi-News expert features (ablation) | Summarization FoE · Summarization average | ROUGE-2: 23.7171 -> 21.9752 | Hurt |
| Use only TFN expert features (ablation) | Sentiment analysis FoE · Sentiment analysis average | Accuracy (%): 91.88% -> 91.81% | No clear effect |
| Use only Poem expert features (ablation) | Sentiment analysis FoE · Sentiment analysis average | Accuracy (%): 91.88% -> 91.75% | No clear effect |
| Use Expert 2 rather than Expert 1 features only (ablation) | MMLU FoE · MMLU overall | Accuracy: 49.85 -> 48.27 | Hurt |
| Use Expert 3 rather than Expert 1 features only (ablation) | MMLU FoE · MMLU overall | Accuracy: 49.85 -> 48.54 | Hurt |
| Use the FoE fuser instead of the best individual expert | 15 open-source LLMs · MMLU overall | Accuracy: 47.35 -> 49.85 | Helped |
| Use FoE instead of the strongest individual expert | Linear metric-combination experts · Text generation evaluation average consistency | Pearson correlation: 0.495 -> 0.517 | Helped |
| Use FrugalFoE to select a subset of experts | CIFAR-100 classification · CIFAR-100 super-class classification test | Experts queried: 37.5% of the experts | Helped |
| Exclude five MMLU categories from fuser training (ablation) | MMLU FoE · MMLU unseen categories | Accuracy: 68.29 -> 68.29; 45.45 -> 47.47; 59.47 -> 55.30; 68.69 -> 64.65; 67.49 -> 66.50 | Mixed |
Surprising findings
- A single expert's embeddings alone suffice to select the correct expert almost as well as using all experts in sentiment and summarization tasks.
- Fusing general 7B LLMs as weak experts on MMLU yields 49.85 overall vs 47.35 for best individual LLM, though well below oracle (51.56).
Limitations
- Weak general-purpose LLM experts (MMLU) cannot match oracle performance and show discrepancies across expert choice.
- FrugalFoE not directly applicable to linear-metric experts in the evaluation task.
- Fuser for all expert subsets needs training unless kNN is used, limiting scalability of combinatorial subsets.
Experimental setup
Datasets: CIFAR-100 Twitter Financial News Sentiment poem_sentiment Data Reviews Sentiment Analysis Auditor Sentiment CNN/DailyMail XSum Multi-News BillSum BigPatent AESLC MMLU
Benchmarks: CIFAR-100 superclass sentiment analysis combined test set summarization ROUGE-2 across six tasks MMLU SummEval Newsroom QAGS-CNN QAGS-XSUM HALL-XSUM
Metrics: accuracy expert selection accuracy ROUGE-2 Pearson correlation
Baselines: Experts Average Confidence-based Fusion Ensemble Oracle Expert FrugalML Best Expert
Builds on: FrugalML mixup A* algorithm
Closest papers by meaning
- Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging (ICML 2025 · similarity 0.85)
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.85)
- MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (AAAI 2024 · similarity 0.84)
- Autonomy-of-Experts Models (ICML 2025 · similarity 0.84)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.84)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.