Research map / Architectures and efficiency / Pruning and efficient tuning

Dynamic Model Merging Made Slim

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DiDi-Merging, a data-free dynamic model-merging method that uses differentiable rank optimization to split parameters between a shared low-rank module and per-task low-rank experts, matching prior dynamic methods at about 1.24x single-model parameters.

Problem

Dynamic model merging stores either a full-size shared model with tiny experts or heavy per-task experts, leading to poor accuracy-efficiency trade-offs; the paper seeks a slim balance between shared and expert parameters without original training data.

Approach

Represent both shared and expert task vectors as low-rank modules whose ranks are learned end-to-end via a smooth tanh-based singular-value truncation with a compression penalty, then recover fidelity with a data-free LoRA refinement against the stored task vectors.

Key results

• On 8 ViT-B/32 vision tasks, DiDi-Merging-S reaches 89.3% average at 1.24x parameters, matching TSV-C (89.5% at 2.08x); DiDi-Merging-M reaches 89.8% at 1.4x, the best overall.
• On Llama-3.1-8B-Instruct LLM merging at 2x parameters, DiDi-Merging-L averages 55.0 vs 52.8 for Twin-Merging and 53.2 for FREE-Merging (+5.4% over prior dynamic methods per the text).
• On multimodal merging at 1.4x-2.0x, DiDi-Merging achieves 99.3% retention on VQA-style video tasks and 101.1% on 7 image tasks, outperforming NaiveMC, MMER and FREE-Merging.

Surprising findings

• Some dynamic merged models (including DiDi-Merging) exceed the fine-tuned individual on several audio and point-cloud tasks, attributed to transferred instruction-following patterns from sister MLLMs.
• Static merging followed by uniform-rank LoRA recovery (e.g., WUDI + uniform LoRA at 84.6%) falls notably short of DiDi-Merging-S (89.3%) at the same budget.

Limitations

• Routing uses an oracle MoE-style router that takes task identity at inference; learned input-based routing is left for future work.
• Find-Optimal-Rank stage runs sequentially per task and is the dominant wall-clock bottleneck.
• Reported LLM gains are for a 2x budget; the 1.24x/1.4x budgets only match baselines on harder settings.

Experimental setup

Datasets: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD MMLU-Pro MMLU-redux GPQA-Diamond GSM8K

Benchmarks: SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD MMLU-Pro MMLU-redux GPQA-Diamond GSM8K

Metrics: accuracy normalized average retention (%) parameter overhead (x single model) BLEU ROUGE-L METEOR CIDEr

Baselines: Weight Averaging TSV-Merging CAT-Merging ISO-CTS WUDI-Merging WEMoE SMILE TALL-Mask+TIES Twin-Merging EMR-Merging FREE-Merging TSV-C

Builds on: TSV-Merging AdaLoRA DyLoRA Dobi-SVD Twin-Merging

Closest papers by meaning

• Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.91)
• Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.90)
• CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging (ICML 2025 · similarity 0.90)
• Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.90)
• Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.89)
• No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.89)