Research map / Architectures and efficiency / Pruning and efficient tuning
RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RobustMerge, a training-free merging method for LoRA modules of multimodal LLMs that prunes small-magnitude parameters, rescales LoRA B by complementary coefficients from A, and normalizes scaling across tasks to maintain singular-vector direction robustness.
Problem
Merging parameter-efficient (LoRA) task modules into one multi-task model degrades badly compared with full fine-tuning merging, and existing methods also rely on seen-task validation data or storage and fail to generalize to unseen tasks.
Approach
Analyzes LoRA merging via SVD and argues that direction robustness of singular vectors matters; prunes small-magnitude LoRA parameters, scales the B matrix with coefficients derived from A's statistics to boost tail singular values, and normalizes these coefficients across tasks, all without data or decomposition.
Key results
- On MM-MergeBench eight seen tasks, RobustMerge averages 57.33 vs 53.93 for Task Arithmetic (about 3.4 points better), and 37.99 vs 33.31 on four unseen tasks.
- On eight CLIP-ViT-B-32 vision tasks, RobustMerge reaches 55.9% average vs 51.5% for Ties-merging (+4.4), and 70.9% with CLIP-ViT-L-14 (+2.0 over Ties).
- Ablation shows pruning and scaling gives +2.21 average over TA, and adding cross-task normalization brings it to +3.40.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply RobustMerge instead of Task Arithmetic | LoRA merging with LLaVA · MM-MergeBench seen tasks | average accuracy: 53.93 -> 57.33 | Helped |
| apply RobustMerge instead of PCB-merging | LoRA merging with LLaVA · MM-MergeBench unseen tasks | average accuracy: 33.53 -> 37.99 | Helped |
| apply RobustMerge instead of Ties-merging | vision-task merging with CLIP-ViT-B-32 · Vision tasks | average accuracy: 51.5 -> 55.9 | Helped |
| apply RobustMerge instead of Ties-merging | vision-task merging with CLIP-ViT-L-14 · Vision tasks | average accuracy: 68.9 -> 70.9 | Helped |
| apply RobustMerge instead of Task Arithmetic | DoRA merging · MM-MergeBench seen tasks | average accuracy: 55.42 -> 56.08 | Helped |
| add pruning and complementary scaling (ablation) | LoRA merging with LLaVA · MM-MergeBench seen tasks | average accuracy: 53.93 -> 56.14 | Helped |
| add cross-task normalization to pruning and complementary scaling (ablation) | LoRA merging with LLaVA · MM-MergeBench seen tasks | average accuracy: 56.14 -> 57.33 | Helped |
| increase the pruning rate (ablation) | RobustMerge · MM-MergeBench | average accuracy | Mixed |
| use magnitude-based pruning instead of aggregated-sign pruning (ablation) | RobustMerge · multimodal tasks | average performance: +2.3% | Helped |
| use magnitude-based pruning instead of random pruning (ablation) | RobustMerge · multimodal tasks | average performance: +4.0% | Helped |
| replace complementary scaling with scaling coefficients from A and B simultaneously (ablation) | RobustMerge · MM-MergeBench seen tasks | average performance: 2.4% decrease | Hurt |
| add cross-task normalization (ablation) | RobustMerge · MM-MergeBench seen tasks | average performance: 1.2% | Helped |
Surprising findings
- Sign-based (Ties-style) pruning performs worse than random pruning in parameter-efficient merging, suggesting sign conflict is not the main issue for LoRA merging unlike full fine-tuning.
- Existing merging methods often underperform zero-shot on PEFT modules, and sometimes even Task Arithmetic does not beat simple baselines.
Limitations
- Only validated on LLaVA and CLIP structures and a limited set of tasks.
- Does not design a specific algorithm on decomposed matrices for efficiency.
- Method is a post-processing heuristic without theoretical guarantees beyond a simplified two-module analysis.
Experimental setup
Datasets: ScienceQA ImageNet VQAv2 RefCOCO OCRVQA Flickr30k VizWiz-caption IconQA ImageNet-R A-OKVQA Screen2Word TabMWP
Benchmarks: MM-MergeBench POPE MME MMBench Cars MNIST EuroSAT GTSRB DTD RESISC45 SUN397 SVHN
Metrics: accuracy average accuracy standard deviation
Baselines: Task Arithmetic Ties-merging DARE PCB-merging LoraHub Multi-Task learning Zero-Shot
Builds on: Task Arithmetic Ties-merging DARE LoRA LLaVA
Closest papers by meaning
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.88)
- EMR-Merging: Tuning-Free High-Performance Model Merging (NeurIPS 2024 · similarity 0.87)
- Less is More: Efficient Model Merging with Binary Task Switch (CVPR 2025 · similarity 0.87)
- Dynamic Model Merging Made Slim (NeurIPS 2026 · similarity 0.87)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.87)
- CALM: Consensus-Aware Localized Merging for Multi-Task Learning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.