Research map / Architectures and efficiency / Pruning and efficient tuning
PLeaS - Merging Models with Permutations and Least Squares
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PLeaS, a two-step model merging method that matches permuted neurons across two same-architecture networks (partial merging to a chosen width) and then fits layer-wise least squares weights, with a data-free variant PLeaS-free using ImageNet.
Problem
Merging fine-tuned models that have the same architecture but come from different initializations and different datasets, into a single model of a chosen size without requiring the training data of the component models.
Approach
Generalizes Git Re-Basin to partial merging (per-layer choice of how many neuron pairs to merge, keeping dissimilar features separate) and sets merged weights via layer-wise least squares that mimic the permuted ensemble of features of the original models, rather than plain weight averaging.
Key results
- On DomainNet at same size (1x), PLeaS averages 38.6% on shared label space vs 30.7% for MuDSC and 25.9% for ZipIt!, and 74.9% vs 68.5% (MuDSC) on different label spaces linear probe.
- PLeaS-free stays within 2% of PLeaS at 1.2x size for different label spaces and drops under 1% at 1x for DomainNet.
- PLeaS approaches ensemble accuracy with 40% fewer parameters, and on same-initialization merging at 1.2x size reaches 75.9 on Real vs ensemble 76.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add layer-wise least-squares fitting to the permutation-matched merged model (ablation) | PLeaS, ResNet-50 · DomainNet, shared label space (Avg) | accuracy: 17.8 -> 38.6 | Helped |
| Add layer-wise least-squares fitting to the permutation-matched merged model (ablation) | PLeaS, ResNet-50 · CUB, Pets, Dogs, NABirds (Avg) | linear probe accuracy: 67.1 -> 74.9 | Helped |
| Use ImageNet data instead of fine-tuning-domain data to compute activations (ablation) | PLeaS-free, ResNet-50 · DomainNet, shared label space | accuracy: less than 1% drop at 1× model size | Hurt |
| Use ImageNet data instead of fine-tuning-domain data to compute activations (ablation) | PLeaS-free, ResNet-50 · Different label spaces | linear probe accuracy: within 2% of the PLeaS at 1.2× model size; less than 4% at 1× model size | Hurt |
| Use weight matching rather than activation matching for permutations (ablation) | PLeaS-Weight, ResNet-50 · Shared and different label spaces | accuracy | No clear effect |
| Increase the merged model size from 1.0 to 1.2 (ablation) | PLeaS, ResNet-50 · DomainNet, shared label space | accuracy: 1.0; 1.2 | Helped |
| Merge models fine-tuned from the same initialization using PLeaS (ablation) | PLeaS, ResNet-50 · Clipart | accuracy: 58.2 -> 59.6 | Helped |
| Merge models fine-tuned from the same initialization using PLeaS (ablation) | PLeaS, ResNet-50 · Infograph | accuracy: 28.9 -> 29.5 | Helped |
| Merge models fine-tuned from the same initialization using PLeaS (ablation) | PLeaS, ViT · CIFAR-50+50 (Joint) | accuracy: 72.6 -> 73.3 | Helped |
Surprising findings
- Simple weight averaging after permutation (Git Re-Basin) is far weaker than expected, and naive RegMean-style least squares without permutations performs poorly when merging models from different initializations.
Limitations
- Limited to merging models with the same architecture.
- Two-stage algorithm has greater running time than some existing methods such as task vector averaging, RegMean and ZipIt!.
- Gains on same-initialization ViT merging are small (under 1 point).
Experimental setup
Datasets: DomainNet CUB NABirds Oxford-IIIT Pets Stanford Dogs CIFAR-100 ImageNet
Benchmarks: DomainNet CUB NABirds Oxford-IIIT Pets Stanford Dogs CIFAR-50+50
Metrics: accuracy linear probe accuracy average accuracy vs model size ratio
Baselines: Simple Averaging (task vectors) RegMean ZipIt! Git Re-Basin MuDSC MoE Ensemble
Builds on: Git Re-Basin RegMean ZipIt! REPAIR
Closest papers by meaning
- Training-Free Pretrained Model Merging (CVPR 2024 · similarity 0.86)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.86)
- Parameter Competition Balancing for Model Merging (NeurIPS 2024 · similarity 0.86)
- Rethinking Layer-wise Model Merging through Chain of Merges (NeurIPS 2026 · similarity 0.86)
- CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging (ICML 2025 · similarity 0.85)
- $C^2M^3$: Cycle-Consistent Multi-Model Merging (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.