Research map / Architectures and efficiency / RNNs and normalization
ZipIt! Merging Models from Different Tasks without Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZipIt!, a training-free method that merges two same-architecture models trained on disjoint tasks by zipping features both within and across models, with partial zipping to yield a multi-head model.
Problem
Merging differently initialized models trained on completely disjoint tasks (disjoint label sets or different datasets) into one multi-task model without any additional training, where permutation-based merging fails.
Approach
Generalize permutation-based merging into a "zip" operation that merges correlated features both within and across the two models via a merge matrix and its unmerge pseudoinverse, propagated through BatchNorm, ReLU, pooling and skip connections, plus partial zipping up to a chosen layer to create a multi-head model.
Key results
- On CIFAR-10 (5+5) with ResNet-20x4, ZipIt! (all layers zipped) reaches 79.1% joint accuracy vs 46.2% for Git Re-Basin and 58.4% for Permute; zipping 13/20 layers gives 83.8% vs ensemble 87.4%.
- On CIFAR-100 (50+50) with ResNet-20x8, ZipIt!_13/20 reaches 70.2% joint accuracy vs 40.9% for Git Re-Basin, close to the 73.5% ensemble.
- On ImageNet-1k (200+200), all-layer ZipIt! and Permute get 8.6% joint accuracy vs 3.1% for Git Re-Basin; partial zipping ZipIt!_10/50 approaches the ensemble.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ZipIt! instead of Permute | ResNet-20×4 model merging · CIFAR-10 (5+5) | joint accuracy: 58.4\conf{6.8} -> 79.1\conf{1.1} | Helped |
| use ZipIt! instead of Permute | ResNet-20×8 model merging · CIFAR-100 (50+50) | joint accuracy: 42.8\conf{0.7} -> 54.9\conf{0.8} | Helped |
| leave the last stage unzipped (ablation) | ZipIt! ResNet-20×4 · CIFAR-10 (5+5) | joint accuracy: 79.1\conf{1.1} -> 83.8\conf{3.1} | Helped |
| leave the last stage unzipped (ablation) | ZipIt! ResNet-20×8 · CIFAR-100 (50+50) | joint accuracy: 54.9\conf{0.8} -> 70.2\conf{0.4} | Helped |
| allow within-model feature merges (ablation) | ZipIt! · CIFAR-100 (50+50) | accuracy | Helped |
| vary the same-model merge budget (ablation) | ZipIt! · ImageNet-1k (200+200) | accuracy | No clear effect |
| increase model width (ablation) | ZipIt! and Permute · CIFAR-100 (50+50), models narrower than 4× | joint accuracy | No clear effect |
| increase model width (ablation) | ZipIt! · CIFAR-100 (50+50) | joint accuracy | Helped |
| use greedy feature matching instead of identity or permutation matching (ablation) | ZipIt! · CIFAR-10 (5+5) | joint accuracy: +21.2% | Helped |
| allow repeated feature matches (ablation) | ZipIt! · matching algorithm evaluation | accuracy | Helped |
| use data augmentation when computing feature matches (ablation) | ZipIt! · CIFAR-100 (50+50) | average per-task accuracy: almost 4% | Helped |
| use data augmentation when computing feature matches (ablation) | ZipIt! · ImageNet-1k (200+200) | accuracy | No clear effect |
Surprising findings
- Same-model merging budget mattered greatly on CIFAR but not on ImageNet, suggesting little redundancy within ImageNet models at this capacity.
- With cross-entropy loss on CIFAR-100, merging the full network fails for all methods, and only partial zipping lets merged models exceed the original models' accuracy.
Limitations
- Greedy matching is used instead of an optimal graph-based solver, which is prohibitively slow (11 minutes for ResNet-20x4).
- Partial zipping leaves unmerged heads, so the merged model keeps more FLOPs and multiple heads rather than a single fully merged network.
- Approximation of passing merge matrices through nonlinear ReLU layers; the theory only holds for random initialization two-layer networks.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k Stanford Dogs Oxford Pets CUB200 NABirds Pascal VOC
Benchmarks: CIFAR-10 (5+5) CIFAR-100 (50+50) ImageNet-1k (200+200) Stanford Dogs Oxford Pets CUB200 NABirds Pascal VOC
Metrics: joint accuracy per-task accuracy mIoU FLOPs
Baselines: Weight Averaging Git Re-Basin Permute Ensemble REPAIR
Builds on: Git Re-Basin REPAIR Model Soups CLIP
Closest papers by meaning
- Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (NeurIPS 2025 · similarity 0.88)
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces (ICML 2025 · similarity 0.88)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.88)
- OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging (ICLR 2026 · similarity 0.87)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging (NeurIPS 2024 · similarity 0.87)
- Rethinking Layer-wise Model Merging through Chain of Merges (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.