Research map / Optimization / Implicit bias and generalization
Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why weight matching (WM) permutations yield linear mode connectivity, attributing it to alignment of large-singular-value singular vectors rather than L2 distance reduction, extends the view to activation matching, and contrasts WM with STE for merging three or more models.
Problem
Why permutation search by weight matching (WM) satisfies linear mode connectivity (LMC) between independently trained networks, given that WM reduces the L2 distance only modestly.
Approach
LMC from WM arises because permutations align the singular vectors with large singular values of each layer across models, so the dominant functional components of hidden-layer outputs match; the L2 distance reduction itself is not the cause. Also, STE does not align singular vectors, so WM generalizes better to three or more models.
Key results
- Permutations found by WM reduce the L2 distance between models by only about 6% to 20% even when LMC holds, and a second-order Taylor approximation fails to predict the barrier.
- With WM, singular-vector alignment R at gamma=0.3 exceeds 0.8 for the MLP between merged and pre-merged models, while R is near zero without WM.
- For three-model merging, the loss barrier between permuted models is lower with WM than STE (e.g., VGG11 0.141 vs 2.172; ResNet20 0.294 vs 1.693).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WM instead of STE to match models before merging three models | VGG11 · CIFAR10 | loss barrier: $2.172 \pm 0.989$ -> $0.141 \pm 0.141$ | Helped |
| Use WM instead of STE to match models before merging three models | VGG11 · CIFAR10 | accuracy barrier: $32.013 \pm 8.193$ -> $10.12 \pm 5.117$ | Helped |
| Use WM instead of STE to match models before merging three models | ResNet20 · CIFAR10 | loss barrier: $1.693 \pm 0.168$ -> $0.294 \pm 0.098$ | Helped |
| Use WM instead of STE to match models before merging three models | ResNet20 · CIFAR10 | accuracy barrier: $34.483 \pm 2.426$ -> $7.23 \pm 0.99$ | Helped |
| Use WM instead of STE to match models before merging three models | MLP · FMNIST | loss barrier: $0.023 \pm 0.118$ -> $-0.174 \pm 0.051$ | Helped |
| Use WM instead of STE to match models before merging three models | MLP · FMNIST | accuracy barrier: $15.97 \pm 1.724$ -> $4.337 \pm 1.434$ | Helped |
| Use WM instead of STE to match models before merging three models | MLP · MNIST | loss barrier: $0.017 \pm 0.014$ -> $-0.031 \pm 0.003$ | Helped |
| Use WM instead of STE to match models before merging three models | MLP · MNIST | accuracy barrier: $2.312 \pm 0.457$ -> $0.475 \pm 0.069$ | Helped |
| Use WM permutation matching rather than no permutation | VGG11 · CIFAR10 | L2 distance: $799.503\pm16.396$ -> $746.465\pm19.576$ | No clear effect |
| Use WM permutation matching rather than no permutation | ResNet20 · CIFAR10 | L2 distance: $710.762\pm16.261$ -> $661.055\pm12.539$ | No clear effect |
| Use WM permutation matching rather than no permutation | MLP · FMNIST | L2 distance: $121.853\pm5.83$ -> $100.041\pm4.71$ | No clear effect |
| Use WM permutation matching rather than no permutation | MLP · MNIST | L2 distance: $81.231\pm5.58$ -> $64.751\pm4.795$ | No clear effect |
Surprising findings
- Second-order Taylor approximation of the barrier fails for WM permutations, since the models are not close enough in L2 distance.
- STE-found permutations yield near-zero singular-vector alignment yet still achieve small two-model barriers, but fail to produce LMC among three models.
Limitations
- Analysis is largely per-layer and does not fully account for the multi-layer structure of networks.
- Empirical results rely on a few architectures and datasets (MLP, VGG11, ResNet20, ResNet50), with the width-dependence of LMC noted as requiring sufficiently wide models.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR10 ImageNet
Benchmarks: MNIST Fashion-MNIST CIFAR10 ImageNet
Metrics: loss barrier accuracy barrier L2 distance reduction rate singular vector alignment R
Baselines: weight matching straight-through estimator activation matching
Builds on: Git Re-Basin Sinkhorn re-basin Entezari et al. permutation invariance conjecture Frankle et al. linear mode connectivity
Closest papers by meaning
- Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries (ICML 2025 · similarity 0.92)
- Generalized Linear Mode Connectivity for Transformers (NeurIPS 2025 · similarity 0.87)
- Linear Mode Connectivity in Differentiable Tree Ensembles (ICLR 2025 · similarity 0.87)
- On Linear Mode Connectivity of Mixture-of-Experts Architectures (NeurIPS 2025 · similarity 0.86)
- $C^2M^3$: Cycle-Consistent Multi-Model Merging (NeurIPS 2024 · similarity 0.85)
- Understanding Mode Connectivity via Parameter Space Symmetry (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.