Research map / Large language models / Language, safety and interpretability
$\texttt{Model-GLUE}$: Democratized LLM Scaling for A Large Model Zoo in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks LLM aggregation techniques (selective merging, model/block/FFN-level mixture, stacking) on Llama-2-based model zoos and proposes Model-GLUE, which clusters mergeable models, merges within clusters, then mixes clusters at model level, reporting a 5.61% average gain without extra training.
Problem
Combining many heterogeneous open-source LLMs (different sizes of model zoo, similarity, and architectures) via merging or mixture often degrades performance, and no comprehensive benchmark or guideline exists for selecting and aggregating such a diverse model zoo.
Approach
Cluster models by weight cosine similarity, selectively merge only mergeable models within each cluster using greedy/evolutionary coefficient search on a proxy dataset, then combine the resulting cluster representatives (plus unmergeable models) via a model-level mixture with a sample-level linear router initialized from prompt vectors.
Key results
- Model-GLUE reaches 54.43% average on the Which16 zoo versus 48.82% for the best single model (about 5.61 points improvement).
- On Which4 (Domain), model-level mixture (43.95%) slightly outperforms merging (43.86%), while on Which4 (Chat) merging (41.54%) clearly beats mixture (39.16%).
- On a Mistral-based Which8 zoo, Model-GLUE improves average to 60.18% versus 57.69% for the best single model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine selected mergeable clusters by model-level mixture | Model-GLUE on Which16 · Which16 | Average: 48.82% -> 54.43% | Helped |
| Use model-level mixture rather than FFN-level or block-level mixture (ablation) | Which2 and Which4 · Which2; Which4 | Average: 37.68% -> 40.04%; 38.70% -> 42.65% | Helped |
| Merge the initial layers before FFN-level mixture (ablation) | F-L-T · Which2; Which4 | Average: 39.20% -> 40.02%; 41.74% -> 43.65% | Helped |
| Replace the MLP router with a linear router (ablation) | FFN-level and block-level mixture · Which2 | Average: 38.78% -> 39.20%; 37.01% -> 37.88% | Helped |
| Use token-level rather than sample-level router input (ablation) | FFN-level mixture · Which2; Which4 Chat | Average: 38.91% -> 39.20%; 39.22% -> 40.85% | Helped |
| Use sample-level rather than token-level router input (ablation) | FFN-level mixture · Which4 Domain | Average: 41.74% -> 42.57% | Helped |
| Apply direct model-level mixture to the unmergeable Llama-2-chat and CrystalChat pair | Model-level mixture · Llama-2-chat and CrystalChat | Average: 45.85% -> 43.77% | Hurt |
| Use selective coefficient search rather than simple average merging (ablation) | Heuristic merging · Which12 | Average: 42.61% -> 43.56% | Helped |
| Use finer-grained coefficient groups of 8 layers rather than 32 (ablation) | Evolutionary linear merging · Which12 | Average: 43.59% -> 43.85% | Helped |
| Use group size 1 rather than group size 8 (ablation) | Evolutionary linear merging · Which12 | Average: 43.85% -> 42.92% | Hurt |
| Merge Llama-2-chat with Vicuna rather than CodeLlama | Model merging · Llama-2-chat plus Vicuna; Llama-2-chat plus CodeLlama | Average: 37.68% -> 40.84%; 40.20% -> 38.09% | Mixed |
| Combine heterogeneous model clusters with Model-GLUE rather than using Full Merging | Which16 aggregation · Which16 | Average: 43.56% -> 54.43% | Helped |
Surprising findings
- Merging Llama-2-chat with CodeLlama (cosine similarity 0.535) degrades performance below the best single model, and TIES drops to 16.13% average.
- Hybrid mixture (merging bottom layers, FFN mixture on top) improves average and reduces inference memory, and the linear router beats the MLP router without extra training.
Limitations
- Benchmarking focuses on empirical results without deep analysis of why merging helps; theoretical understanding is limited.
- Model stacking and model communication are only preliminarily discussed and not fully integrated into Model-GLUE.
- Evaluated mainly on Llama-2-based 7B models; the cosine-similarity clustering threshold (0.95) is chosen heuristically from limited examples.
Experimental setup
Datasets: GPT4All tinybenchmarks subsets MBPP validation set HumanEvalPack JavaScript (20% split)
Benchmarks: ARC WinoGrande MMLU GSM8K MBPP HumanEval
Metrics: accuracy average accuracy
Baselines: Linear merging SLERP Model Stock Task Arithmetic DARE TIES F-L-T Mixture Full Merging
Builds on: Model Soup Task Arithmetic TIES DARE Branch-Train-MiX
Closest papers by meaning
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.92)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.91)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.91)
- Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging (NeurIPS 2025 · similarity 0.90)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.90)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.