Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces W-Asymmetric and sigma-Asymmetric (FiGLU) networks that remove parameter space symmetries, and empirically studies their effects on linear mode connectivity, Bayesian training, metanetworks, and monotonic linear interpolation.
Problem
Parameter space symmetries in neural networks confound the analysis of phenomena such as linear mode connectivity, model merging, Bayesian inference, and loss-landscape geometry, and are hard to study theoretically; the paper asks what their empirical impact is.
Approach
Modify standard architectures to reduce parameter symmetries while keeping them trainable with standard gradient methods: (1) W-Asym fixes a random binary-masked subset of each linear layer's weights to constant Gaussian values so rows are distinct, breaking computation-graph automorphisms; (2) sigma-Asym replaces elementwise nonlinearities with FiGLU, a non-elementwise gated unit with a fixed ran
Key results
- W-Asym MLP on MNIST reaches a midpoint test loss barrier of -0.012 without alignment, versus 0.188 for standard networks and -0.006 for Git Re-Basin.
- All 300 W-Asym ResNets satisfy monotonic linear interpolation (100% vs 26.3% for standard ResNets).
- Bayesian 16-layer MLPs on CIFAR-10 fail to train (13.54% test accuracy) while W-Asym 16-layer MLPs reach 55.16%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use W-Asymmetric weights instead of standard weights | MLP · MNIST midpoint interpolation | midpoint test loss barrier: 0.188 ± .12 -> -0.012 ± .00 | Helped |
| Use σ-Asymmetric FiGLU instead of standard network | MLP · MNIST midpoint interpolation | midpoint test loss barrier: 0.188 ± .12 -> 0.117 ± .01 | Helped |
| Use W-Asymmetric weights instead of Git-ReBasin alignment | ResNet · CIFAR-10 midpoint interpolation | midpoint test loss barrier: 2.041 ± .21 -> 0.934 ± .72 | Helped |
| Use σ-Asymmetric FiGLU instead of standard network | ResNet · CIFAR-10 midpoint interpolation | midpoint test loss barrier: 3.287 ± .32 -> 2.521 ± .46 | Helped |
| Use W-Asymmetric Bayesian network instead of standard Bayesian network | MLP-16 · CIFAR-10 | Test Acc: 13.54 ± 2.0 -> 55.16 ± .44 | Helped |
| Use W-Asymmetric Bayesian network instead of standard Bayesian network | ResNet20 · CIFAR-10 | Test Acc: 81.98± 1.2 -> 81.94± 0.6 | No clear effect |
| Use W-Asymmetric Bayesian network instead of standard Bayesian network | ResNet20 (BN) · CIFAR-100 | Test Acc: 56.83 ± .62 -> 58.40 ± .62 | Helped |
| Train metanetwork on W-Asymmetric ResNets instead of standard ResNets | MLP metanetwork · CIFAR-10 test-accuracy prediction | R²: .330±.04 -> .594±.12 | Helped |
| Train metanetwork on W-Asymmetric ResNets instead of standard ResNets | DMC metanetwork · CIFAR-10 test-accuracy prediction | R²: .950±.01 -> .967±.01 | Helped |
| Train metanetwork on W-Asymmetric ResNets instead of standard ResNets | DeepSets metanetwork · CIFAR-10 test-accuracy prediction | R²: .855±.01 -> .936±.00 | Helped |
| Train metanetwork on W-Asymmetric ResNets instead of standard ResNets | StatNN metanetwork · CIFAR-10 test-accuracy prediction | R²: .976±.00 -> .978±.00 | Helped |
| Use σ-Asymmetric FiGLU instead of standard network | ResNet · CIFAR-10 initialization-to-training interpolation | Percent Monotonic: 26.3% -> 87.3% | Helped |
Surprising findings
- Asymmetric networks interpolate well without any permutation alignment, even though their parameter-space distance is similar to standard networks.
- sigma-Asym (FiGLU) networks break symmetries less effectively than W-Asym networks, and several FiGLU variants tried did not work well.
Limitations
- sigma-Asym networks do not remove symmetries as effectively as W-Asym networks and do not match Git Re-Basin in linear mode connectivity
- Universal approximation proven only for W-Asym MLPs, not sigma-Asym; scale symmetry removal for W-Asym not formally proven
- Asymmetric networks train more slowly as asymmetry increases
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 ogbn-arXiv
Benchmarks: MNIST CIFAR-10 CIFAR-100 ogbn-arXiv
Metrics: midpoint test loss barrier negative log likelihood expected calibration error test accuracy R^2 Kendall tau percent monotonic local convexity global convexity
Baselines: Git Re-Basin standard MLP standard ResNet DMC DeepSets StatNN
Builds on: Git Re-Basin neural DAG automorphisms SiLU / Swish GLU DeepSets
Closest papers by meaning
- Parameter symmetries determine representational geometry in overparameterized nonlinear networks (NeurIPS 2026 · similarity 0.89)
- Understanding Mode Connectivity via Parameter Space Symmetry (ICML 2025 · similarity 0.88)
- Symmetries in Overparametrized Neural Networks: A Mean Field View (NeurIPS 2024 · similarity 0.88)
- A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (NeurIPS 2025 · similarity 0.88)
- Scale Equivariant Graph Metanetworks (NeurIPS 2024 · similarity 0.87)
- Symmetry Induces Structure and Constraint of Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.