Research map / Vision-language and multimodal / Multimodal LLMs

OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces an MLLM model-merging benchmark (capabilities VQA/Geometry/Chart/OCR/Grounding and vision-audio-video modalities) and OptMerge, a data-free merging method that denoises task vectors via low-rank approximation and optimizes the merged vector with SGD.

Problem

Model merging for multimodal LLMs has lacked a benchmark that cleanly separates MLLM capability training from evaluation, and prior merging methods for MLLMs are either limited to two models or require test data or hyperparameter search.

Approach

Extend WUDI-style data-free optimization of the merged task vector with centering, truncated-SVD low-rank approximation of task vectors used as surrogate inputs, SGD for LoRA models, and mean-of-task-vectors initialization, plus a benchmark of task-specialized MLLM checkpoints and modality-merged Omni-style models.

Key results

• OptMerge reaches 63.30 average on Qwen2-VL (LoRA) capability merging vs. 62.23 for Qwen2-VL-Instruct and 58.65 for WUDI Merging.
• On InternVL2.5 full fine-tuning, OptMerge averages 57.44, above mixture training at 57.66 is not exceeded but it beats WUDI Merging's 57.00 and all other merging baselines.
• On modality merging (vision, audio, video with Vicuna-7B), OptMerge averages 67.00 on AVQA and MUSIC-AVQA, above the best individual vision model's 64.11 average.

Surprising findings

• Iso-C collapses on the LoRA-tuned Qwen2-VL (26.69 average) because LoRA task vectors are already low-rank and singular-value averaging shrinks their norm.
• Merging SFT models with excessive post-training (Qwen2.5-Math with Qwen2.5-Coder) yields poor results, which the authors attribute to parameter drift rather than merging algorithm flaws.

Limitations

• Experiments limited to models of around 7B parameters due to resource constraints.
• Public collected datasets may contain lower-quality data.

Experimental setup

Datasets: GQA VQAv2 OKVQA LLaVA-Instruct CogVLM GeoQA+ G-LLaVA ChartQA DVQA OCRVQA TextCaps DocVQA

Benchmarks: VizWiz GQA MathVista MATH-Vision ChartQA TextVQA OCRVQA RefCOCO RefCOCO+ RefCOCOg MUSIC-AVQA AVQA

Metrics: accuracy

Baselines: Weight Averaging Task Arithmetic TIES-Merging DARE TSV Merging Iso-C WUDI Merging NaiveMC DAMC

Builds on: Task Arithmetic WUDI Merging TIES-Merging DARE TSV Merging

Closest papers by meaning

• MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.94)
• AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization (CVPR 2025 · similarity 0.92)
• Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.91)
• Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices (CVPR 2025 · similarity 0.90)
• Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.90)
• EMR-Merging: Tuning-Free High-Performance Model Merging (NeurIPS 2024 · similarity 0.90)