Research map / Architectures and efficiency / RNNs and normalization
Do We Really Need Permutations? Impact of Width Expansion on Linear Mode Connectivity
Akira Ito, Masanori Yamada, Daiki Chijiwa, Atsutoshi Kumagai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add width expansion to independently trained models (no permutation) | weight-averaged merged model · MNIST/FMNIST/CIFAR-10 | merged test accuracy | Improved |
| add inverse-temperature calibration of the softmax | merged model of wide networks · MNIST/FMNIST/CIFAR-10 | test loss barrier | Improved |
| add weight-matching permutation to wide merged MLP (ablation) | MLP merge · FMNIST | accuracy barrier (%): 2.467 -> 4.925 | Hurt |
| add weight-matching permutation to wide merged VGG-11 (ablation) | VGG-11 merge · CIFAR-10 | accuracy barrier (%): 1.308 -> 7.000 | Hurt |
| add weight-matching permutation to wide merged ResNet-20 (ablation) | ResNet-20 merge · CIFAR-10 | accuracy barrier (%): 2.694 -> 5.135 | Hurt |
| add weight-matching permutation to wide merged MLP (ablation) | MLP merge · MNIST | accuracy barrier (%): 0.519 -> -0.027 | No clear effect |
| add weight-matching permutation to wide merged VGG-11 (ablation) | VGG-11 merge · CIFAR-10 | calibrated loss barrier: 0.066 -> 0.177 | Hurt |
| add weight-matching permutation to wide merged ResNet-20 (ablation) | ResNet-20 merge · CIFAR-10 | calibrated loss barrier: 0.087 -> 0.173 | Hurt |
| add random permutation before merging wide models (ablation) | merged wide models · lambda=1/2 | test accuracy | No clear effect |
| add weak weight decay 10^-4 instead of 0.003 (ablation) | VGG-11 and ResNet-20 merge · CIFAR-10 | merge barrier / LEWC | Hurt |
Closest papers by meaning
- Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods (ICLR 2025 · similarity 0.90)
- Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries (ICML 2025 · similarity 0.90)
- Generalized Linear Mode Connectivity for Transformers (NeurIPS 2025 · similarity 0.87)
- Linear Mode Connectivity in Differentiable Tree Ensembles (ICLR 2025 · similarity 0.87)
- On Linear Mode Connectivity of Mixture-of-Experts Architectures (NeurIPS 2025 · similarity 0.85)
- Understanding Mode Connectivity via Parameter Space Symmetry (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
