Research map / Architectures and efficiency / RNNs and normalization
Layer-wise linear mode connectivity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces layer-wise linear mode connectivity, showing that averaging individual layers of deep networks produces no averaging barrier, with a convexity proof for deep linear networks and a robustness-based explanation.
Problem
Model averaging (e.g., in federated learning) can create loss barriers at the midpoint between independently trained networks; the layer-wise structure of these barriers is poorly understood for deep networks.
Approach
Defines layer-wise linear interpolation instability: interpolate only one layer (or cumulative groups of layers) between two models while keeping others fixed, and measure the midpoint loss. Shows deep networks have no single-layer averaging barriers, proves layer-wise convexity for deep linear networks, and links barriers to robustness and special averaging directions.
Key results
- Layer-wise averaging of individual ResNet18 layers on CIFAR-10 shows no averaging barrier at alpha=0.5 for any layer, while full-model averaging shows a linear barrier.
- Cumulative averaging shows barriers arising from middle layers, not from the shallowest or deepest layers alone.
- In federated personalization experiments, partial aggregation strategies (body, classifier, critical, middle) perform nearly identically, offering no clear benefit over one another.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use partial layer averaging instead of full averaging | ResNet18 local models · CIFAR-10 non-iid | test accuracy: 61.82; 61.73; 50.31; 51.11; 50.69; 50.59; 50.38 | No clear effect |
| Use partial layer averaging instead of full averaging | ResNet18 local models · CIFAR-10 pathological non-iid | test accuracy: 24.24; 91.72; 93.9; 94.12; 94.52; 94.16; 92.37 | Helped |
| Use full averaging instead of local training | ResNet18 local models · CIFAR-10 pathological non-iid | test accuracy: 91.75 vs. 24.24 | Hurt |
| Use partial layer averaging instead of full averaging | ResNet18 local models · CIFAR-100 pathological non-iid | test accuracy: 18.43; 56.46; 55.12; 55.22; 49.01; 50.42; 55.87 | Mixed |
| Use full averaging instead of local training | ResNet18 local models · DomainNet feature shift | test accuracy: 23.04 -> 27.36 | Helped |
| Use partial layer averaging instead of full averaging | ResNet18 local models · DomainNet feature shift | test accuracy: 27.36; 22.12; 15.73; 14.45; 17.11; 17.43; 11.97 | Hurt |
| Use partial layer averaging instead of full averaging | ResNet18 local models · DomainNet feature shift | test loss: 11.25; 10.54; 9.53; 12.15; 8.06; 10.94; 9.03 | Mixed |
Surprising findings
- Averaging a single shallow layer of ResNet18 can produce a high loss while other single-layer averages do not, and averaging only the first layer leads to high loss in the 1D linear example.
- Partial layer-wise aggregation in personalized federated learning does not significantly differ from alternative partial aggregation schemes, undermining the use of LLMC to pick layers.
Limitations
- Theoretical layer-wise convexity is proven only for deep linear networks; non-linear networks are supported empirically only.
- Analysis is limited to small-scale models and datasets (ResNet18, VGG11, small GPT, ViTs on CIFAR-10), and the averaging barrier is measured at a single midpoint with a 2% noise threshold.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Wikitext MNIST DomainNet
Benchmarks: CIFAR-10 CIFAR-100 Wikitext MNIST DomainNet
Metrics: training loss barrier at midpoint training error barrier test accuracy of local models in federated setup test loss robustness to perturbations
Baselines: full-model averaging local training FedAvg-style partial aggregation (body, classifier)
Builds on: linear mode connectivity Git Re-Basin layer criticality
Closest papers by meaning
- On Linear Mode Connectivity of Mixture-of-Experts Architectures (NeurIPS 2025 · similarity 0.86)
- Understanding Mode Connectivity via Parameter Space Symmetry (ICML 2025 · similarity 0.83)
- Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks (ICLR 2025 · similarity 0.83)
- Rethinking Layer-wise Model Merging through Chain of Merges (NeurIPS 2026 · similarity 0.83)
- Linear Mode Connectivity in Differentiable Tree Ensembles (ICLR 2025 · similarity 0.82)
- On the Nonlinearity of Layer Normalization (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.