Research map / Vision-language and multimodal / Multimodal LLMs
AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaMMS, a model merging method for heterogeneous 7B MLLMs that maps parameters across different architectures, linearly interpolates them, and selects the interpolation coefficient unsupervised via generation consistency on 100 unlabeled inputs.
Problem
Model merging is hard for multimodal LLMs because their architectures differ (extra duplicated multimodal weights, asymmetric parameter spaces), and existing merging methods need labeled validation data to pick hyperparameters.
Approach
A parameter mapping function aligns duplicated multimodal weights between heterogeneous MLLMs, followed by linear interpolation of mapped weights, with the coefficient alpha chosen by minimizing response differences between adjacent alpha candidates (generation consistency) rather than by labeled accuracy.
Key results
- On merging LLaVA-OneVision-7B into Qwen2-VL-7B, AdaMMS reaches a SUM of 563.56 vs 559.07 for the base Qwen2-VL and 536.32 for Task Arithmetic, the best baseline sum being MetaGPT at 560.75.
- The unsupervised selection picks an alpha within 0.5 points of the oracle best per task on the Qwen2-VL pair.
- Across six model pairs in the appendix, AdaMMS has the highest total performance gain (+91.92) over the averaged original models, ahead of Task Arithmetic (+47.45).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use unsupervised generation-consistency search for the interpolation coefficient | AdaMMS merging LLaVA-OneVision into Qwen2-VL · Qwen2-VL + LLaVA-OneVision, 8-task SUM | SUM: 563.56 vs 560.75 | Helped |
| use unsupervised generation-consistency search for the interpolation coefficient | AdaMMS merging LLaVA-v1.5 into CogVLM · CogVLM + LLaVA-v1.5, 8-task SUM | SUM: 459.21 vs 445.97 | Helped |
| use an exact-match difference metric instead of sentence-embedding differences during coefficient search (ablation) | AdaMMS merging LLaVA-OneVision into Qwen2-VL · Qwen2-VL + LLaVA-OneVision, MME_sum | score: 83.36 vs 82.36 | No clear effect |
| search with a 100-input subset instead of the full test set (ablation) | AdaMMS merging LLaVA-OneVision into Qwen2-VL · Qwen2-VL + LLaVA-OneVision, 8-task SUM | SUM: 563.70 vs 562.12 | No clear effect |
| use sentence-embedding differences with a 100-input subset instead of exact-match differences on the full set (ablation) | AdaMMS merging LLaVA-OneVision into Qwen2-VL · Qwen2-VL + LLaVA-OneVision, 8-task SUM | SUM: 563.56 vs 563.70 | No clear effect |
| vary the interpolation coefficient from 0.1 to 0.6 (ablation) | AdaMMS linear interpolation merging LLaVA-OneVision into Qwen2-VL · Qwen2-VL + LLaVA-OneVision, OCRBench | score: 85.50 -> 71.30 | Hurt |
| search with granularity 0.1 rather than finer coefficient steps (ablation) | AdaMMS · Qwen2-VL + LLaVA-OneVision, MME and OCRBench | performance | No clear effect |
| merge the weaker model into the base model on tasks with a large original performance gap | model merging methods including AdaMMS · OCRBench and TextVQA | score | Hurt |
| use AdaMMS rather than the original Cambrian model after merging Yi-VL | Cambrian · OCRBench | score: 58.70 -> 59.20 | Helped |
| use AdaMMS rather than the original Cambrian model after merging Yi-VL | Cambrian · MME | score: 72.50 -> 74.07 | Helped |
Surprising findings
- Merging can hurt when the two original models differ greatly in performance; on OCRBench and TextVQA all merging methods, including AdaMMS, drop below the stronger base model.
- Merging with alpha of 0.7 or higher often collapses language ability, so the search range was restricted to [0, 0.6].
Limitations
- Mapping is only defined for MLLMs built on the same pretrained language model with extra multimodal weights; models from different base LLMs are not handled.
- Performance drops on benchmarks where the two original models have a large performance gap.
- Baseline comparisons rely on a limited set of 7B model pairs and a single alpha search granularity in the main experiments.
Experimental setup
Datasets: MMMU MME SeedBench OCRBench TextVQA OKVQA GQA VizWiz ScienceQA
Benchmarks: MMMU MME SeedBench OCRBench TextVQA OKVQA GQA VizWiz ScienceQA
Metrics: accuracy normalized sum score performance gain over average of original models
Baselines: Task Arithmetic Ties-Merging DARE-Linear DARE-Ties MetaGPT
Builds on: Task Arithmetic Ties-Merging DARE MetaGPT FuseLLM
Closest papers by meaning
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.92)
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.91)
- Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging (NeurIPS 2025 · similarity 0.89)
- Activation-Informed Merging of Large Language Models (NeurIPS 2025 · similarity 0.89)
- Channel Merging: Preserving Specialization for Merged Experts (AAAI 2025 · similarity 0.88)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.