Research map / Architectures and efficiency / Pruning and efficient tuning
Training-Free Pretrained Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MuDSC, a training-free model merging method that matches units by linearly combining weight-space and activation-space similarity matrices, with group-structure adaptations for MHA and GroupNorm.
Problem
Merging multiple single-task models into one multi-task model without extra training and without requiring a shared pre-trained initialization; existing unit matching methods use only weight or only activation similarity, which are inconsistent with each other.
Approach
Match units by a convex combination of activation-space and weight-space similarity matrices (dual-space constraints), solved via an iterative alignment; also adapt matching algorithms (alignment and zip) to group structures such as multi-head attention and group normalization.
Key results
- On ResNet20-GN CIFAR-100 (50+50), MuDSC-Align improves joint accuracy over the second-best method by 8.41%.
- On Taskonomy heterogeneous merging, MuDSC-Zip reaches 'Segment Semantic' scaled performance 36.22 vs 32.51 for Weight Zip and 2.69 gain in Edge2D over Zipit-style baselines.
- On ImageNet 200+200 with DINO-pretrained ResNet-50, MuDSC-Align achieves joint accuracy 44.87 vs 44.85 for Git Re-Basin, with small gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MuDSC dual-space constraints to Zip operation | ResNet20 merging · CIFAR100(50+50) | joint accuracy: 54.69 -> 56.01 | Helped |
| Apply MuDSC dual-space constraints to Zip operation | ResNet20 merging · CIFAR10(5+5) | joint accuracy: 82.44 -> 83.09 | Helped |
| Apply MuDSC dual-space constraints to Zip operation | ResNet20GN merging · CIFAR100(50+50) | joint accuracy: 29.93 -> 30.05 | Helped |
| Apply MuDSC dual-space constraints to alignment-based matching | ImageNet-pretrained ResNet26 merging · CIFAR100(50+50) | joint accuracy: 61.91 -> 62.84 | Helped |
| Apply MuDSC dual-space constraints to Zip operation | ImageNet-pretrained ResNet50GN merging · CIFAR100(50+50) | joint accuracy: 74.52 -> 74.71 | Helped |
| Apply MuDSC dual-space constraints to Zip operation | ImageNet-pretrained ViT-Small merging · CIFAR100(50+50) | joint accuracy: 70.16 -> 70.10 | No clear effect |
| Apply MuDSC dual-space constraints to alignment-based matching | DINO-pretrained ResNet-50 merging · ImageNet400(200+200) | joint accuracy: 44.86 -> 44.87 | Helped |
| Apply MuDSC dual-space constraints to alignment-based matching | Taskonomy encoders · Taskonomy total average | scaled performance: 76.64 -> 77.37 | Helped |
| Apply MuDSC dual-space constraints to Zip operation | Taskonomy encoders · Taskonomy total average | scaled performance: 83.86 -> 84.84 | Helped |
Surprising findings
- Activation-based alignment performs even worse than direct averaging when merging fine-tuned models from the same pretrained initialization.
- Weight-based matching results are identical to plain averaging for fine-tuned pretrained models, since fine-tuned weights are already highly similar.
Limitations
- Group alignment has quadratic complexity in external matching.
- Grouping strategy for group zip is not efficiently solved and is delegated to group alignment results.
- Balance factor alpha must be searched per model pair on training data.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k Taskonomy MNIST
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k Taskonomy
Metrics: joint accuracy per-task accuracy scaled performance
Baselines: Git Re-Basin Activation Align ZipIt Weight Zip Simple averaging
Builds on: Git Re-Basin ZipIt Task Arithmetic
Closest papers by meaning
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.90)
- Whoever Started the interference Should End It: Guiding Data-Free Model Merging via Task Vectors (ICML 2025 · similarity 0.88)
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.88)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.87)
- Representation Surgery for Multi-Task Model Merging (ICML 2024 · similarity 0.87)
- Parameter Competition Balancing for Model Merging (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.