Research map / Optimization / Implicit bias and generalization
Symmetries in Overparametrized Neural Networks: A Mean Field View
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Develops a mean-field theory for overparametrized shallow (multi-layer unit) networks trained under compact-group data symmetries, showing data augmentation, feature averaging and equivariant architectures correspond to Wasserstein gradient flows that preserve weakly and strongly invariant parameter laws.
Problem
How symmetry-leveraging techniques (data augmentation, feature averaging, equivariant architectures) affect the learning dynamics of overparametrized neural networks under distributional data symmetries for a compact group, in the infinite-width limit.
Approach
Lifts shallow networks to Wasserstein gradient flows on parameter measures and defines weakly invariant (G-invariant measures) and strongly invariant (supported on group-fixed parameters) laws; shows DA, FA and freely trained models share the same mean-field dynamics for symmetric data and that strongly invariant laws are preserved by vanilla SGD in the limit.
Key results
- As N grows, vanilla SGD from strongly invariant initialization stays within the strongly invariant set under a weakly invariant teacher, matching the theory
- DA and FA remain within the equivariant subspace almost independently of N
- Vanilla training converges toward DA and FA end-of-training distributions as N grows
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Data Augmentation instead of Feature Averaging | Mean-field training with symmetric initialization · General compact-group symmetry setting | Wasserstein gradient-flow trajectory | No clear effect |
| Use vanilla training instead of Data Augmentation or Feature Averaging | Mean-field training with equivariant data and weakly invariant initialization · Equivariant data | Wasserstein gradient-flow trajectory | No clear effect |
| Constrain the model to strongly invariant parameter distributions | Equivariant-data learning · General compact-group symmetry setting | Population risk optimum | Hurt |
| Initialize in the strongly invariant parameter subspace and freely train | Vanilla training with a weakly invariant teacher · Synthetic teacher-student experiment | Distance to projected parameter distribution | No clear effect |
| Initialize in the strongly invariant parameter subspace and freely train | Vanilla training with an arbitrary teacher · Synthetic teacher-student experiment | Distance to projected parameter distribution | Hurt |
| Increase the number of particles | Vanilla, Data Augmentation, and Feature Averaging training with a weakly invariant teacher · Synthetic teacher-student experiment | Distance between training schemes | No clear effect |
| Initialize the mean-field flow with a weakly invariant distribution | Wasserstein gradient flow of an invariant functional · General compact-group symmetry setting | Weak invariance during training | Helped |
| Initialize in the strongly invariant parameter subspace | Freely trained mean-field flow with invariant risk and regularizer · General compact-group symmetry setting | Strong invariance during training | Helped |
Surprising findings
- Freely trained vanilla SGD from strongly invariant initialization preserves equivariant (SI) structure in the mean-field limit, despite unconstrained parameter updates
- Strongly invariant laws need not contain the population risk minimizer (counterexample)
Limitations
- Relies on technical assumptions (bounded differentiable activations, infinite i.i.d. data) and infinite-width limit
- Experiments limited to small synthetic setting with finite group C2 and N up to 5000
- Results shown for shallow models; extension to general deep networks left open
Experimental setup
Datasets: synthetic teacher-student data
Metrics: Relative Measure Distance (RMD) Wasserstein-2 distance
Baselines: vanilla SGD data augmentation feature averaging equivariant architecture
Builds on: mean-field theory of shallow neural networks Wasserstein gradient flows Mei et al. symmetric WGF analysis of two-layer ReLU networks
Closest papers by meaning
- Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent (NeurIPS 2024 · similarity 0.89)
- On the hardness of learning under symmetries (ICLR 2024 · similarity 0.88)
- Parameter symmetries determine representational geometry in overparameterized nonlinear networks (NeurIPS 2026 · similarity 0.88)
- A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (NeurIPS 2025 · similarity 0.88)
- The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof (NeurIPS 2024 · similarity 0.88)
- Symmetry Induces Structure and Constraint of Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.