Research map / Optimization / Implicit bias and generalization
Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that two-model permutation search (WM, STE) fails to bring multiple models into one linearly connected basin, and proposes STE-MM, a straight-through permutation search for multiple models with an accelerated LAP solver, evaluated on MLP/VGG-11/ResNet-20 trained on MNIST/FMNIST/CIFAR-10.
Problem
Whether linear mode connectivity modulo permutation symmetries, which holds for two independently trained networks, extends to three or more models at once, so that all of them lie in a single approximately convex low-loss basin.
Approach
Extends the straight-through estimator of Ainsworth et al. to n models by optimizing dummy parameters with a random convex combination of all permuted models as the loss, and speeds up the weight-matching LAP step by reusing dual variables from previous iterations.
Key results
- STE-MM keeps merged-model test accuracy and loss close to the originals with up to 10 models, while WM, STE, MergeMany and CCMM degrade as the number of models grows (Figure 1, Figure 3).
- With STE-MM the sharpness of the merged model decreases as the number of models increases for VGG-11 and ResNet-20, whereas MergeMany and CCMM sharpness increases.
- The accelerated WM speeds up processing by roughly 10x for WM and STE, and about 40x for ResNet-20 permutation search.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STE-MM instead of two-model STE or weight matching as the number of merged models increases | Model merging · CIFAR-10, VGG-11 and ResNet-20 | test accuracy | Mixed |
| Use STE-MM instead of MergeMany or CCMM for merging multiple models | Model merging · MNIST, FMNIST and CIFAR-10 | test accuracy | Helped |
| Use STE-MM instead of MergeMany or CCMM for merging multiple models | Model merging · MNIST, FMNIST and CIFAR-10 | test loss difference from the average original-model loss | Helped |
| Increase the number of merged models using STE-MM | STE-MM · MNIST, FMNIST and CIFAR-10 | test loss difference from the average original-model loss | Mixed |
| Use STE-MM to find permutations for model pairs | STE-MM · MNIST, FMNIST and CIFAR-10 | test loss difference for merged pairs | No clear effect |
| Increase model width (ablation) | Model merging methods · CIFAR-10, VGG-11 and ResNet-20 | test loss difference between the merged model and the average original-model los | Helped |
| Increase the number of merged models using STE-MM | STE-MM · MNIST and FMNIST, MLP | sharpness | Helped |
| Increase the number of merged models using STE-MM | STE-MM · CIFAR-10, VGG-11 and ResNet-20 | sharpness | Helped |
| Use MergeMany or CCMM instead of STE-MM as the number of models increases | Model merging · CIFAR-10, VGG-11 and ResNet-20 | sharpness | Hurt |
Surprising findings
- Conventional two-model permutation methods (WM, STE) degrade merged-model accuracy as more models are added, so pairwise LMC does not imply a shared convex basin for multiple models.
- Merged-model sharpness decreases with more models under STE-MM, the opposite trend from MergeMany and CCMM.
Limitations
- Experiments are limited to small MLP, VGG-11, and ResNet-20 models on simple image datasets.
- LMC for multiple models generally requires increasing model width (4x to 16x here), and it is unclear whether it holds without expansion.
- The reason permutations satisfying multi-model LMC exist is not theoretically explained.
Experimental setup
Datasets: MNIST FMNIST CIFAR-10
Benchmarks: MNIST FMNIST CIFAR-10
Metrics: test accuracy test loss loss barrier trace of Hessian (tr(H ⊙ θθ^T)) sharpness L2 distance ratio angle between models
Baselines: WM STE MergeMany CCMM
Builds on: Ainsworth et al. 2023 (straight-through estimator and weight matching) Entezari et al. 2022 (permutation invariance conjecture) MergeMany CCMM
Closest papers by meaning
- Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods (ICLR 2025 · similarity 0.92)
- Generalized Linear Mode Connectivity for Transformers (NeurIPS 2025 · similarity 0.89)
- On Linear Mode Connectivity of Mixture-of-Experts Architectures (NeurIPS 2025 · similarity 0.86)
- Linear Mode Connectivity in Differentiable Tree Ensembles (ICLR 2025 · similarity 0.86)
- Understanding Mode Connectivity via Parameter Space Symmetry (ICML 2025 · similarity 0.86)
- Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.