Research map / Architectures and efficiency / RNNs and normalization
Understanding Mode Connectivity via Parameter Space Symmetry
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses parameter-space symmetry and topology of symmetry groups to count connected components of minima of linear networks, prove mode connectivity up to permutation, show linear mode connectivity can fail, and derive symmetry-induced curves with curvature bounds on linear interpolation barriers.
Problem
Theoretical explanation of mode connectivity (minima connected by low-loss curves) and when linear mode connectivity holds or fails, which remains largely conjectural.
Approach
Link topology of symmetry groups (e.g., GL_h, R+) to the topology of the loss minima to count connected components and bound orbit components; use continuous symmetries to generate explicit constant-loss curves between minima and bound their curvature, giving conditions for approximate linear mode connectivity.
Key results
- Minimum of a multi-layer full-rank linear network has 2^(l-1) connected components, independent of width.
- For a 3-layer 1x1 linear net, the minimum has 4 components without a skip connection and 3 with a residual connection.
- Empirically, loss on symmetry-induced curves stays lower than loss on the linear interpolation between endpoints across 100 random two-layer networks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a skip connection | 1D three-layer linear network · Minimum set | connected components: 4 -> 3 | Helped |
| Apply a suitable neuron permutation before comparing minima | Full-rank linear networks with hidden dimension h ≥ 2 · Linear network minimum | mode connectivity | Helped |
| Use homogeneous-activation rescaling to choose two minima in the same connected component | Multi-layer regression with homogeneous activation · Linear interpolation between minima | loss barrier: arbitrarily large | Hurt |
| Restrict the symmetry orbit to bounded parameters | Homogeneous-activation network · Linear interpolation between points in an orbit | loss barrier | Helped |
| Use symmetry-induced curves instead of linear interpolation | Two-layer leaky-ReLU network · Curves between symmetry-related minima | loss | Helped |
| Bound the curvature of a connecting curve | Loss level set · Straight segment between two minima | loss barrier | Helped |
Surprising findings
- Linear mode connectivity can fail with unbounded error barrier even between minima in the same connected component when a homogeneous activation rescaling symmetry is present.
Limitations
- Exact component counts are derived for linear or full-rank settings; nonlinear extension is left open.
- Empirical validation is limited to small synthetic two-layer networks.
Experimental setup
Datasets: synthetic random data for two-layer sigmoid networks synthetic Gaussian data for leaky ReLU two-layer network
Metrics: loss barrier loss along symmetry-induced curves component counts
Builds on: mode connectivity (Garipov et al.) linear mode connectivity (Frankle et al.) permutation symmetry alignment (Ainsworth et al.)
Closest papers by meaning
- Generalized Linear Mode Connectivity for Transformers (NeurIPS 2025 · similarity 0.91)
- Input Space Mode Connectivity in Deep Neural Networks (ICLR 2025 · similarity 0.89)
- The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof (NeurIPS 2024 · similarity 0.88)
- On Linear Mode Connectivity of Mixture-of-Experts Architectures (NeurIPS 2025 · similarity 0.87)
- Parameter symmetries determine representational geometry in overparameterized nonlinear networks (NeurIPS 2026 · similarity 0.87)
- A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.