Research map / Optimization / Implicit bias and generalization
Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that SGD gradient noise drives parameters along symmetry-induced degenerate directions to unique attractive "noise equilibria," explaining progressive sharpening/flattening, warmup, and aligned representations.
Problem
How gradient noise in SGD biases training dynamics along degenerate directions created by continuous symmetries of the loss, where GD and SGD converge to different solutions.
Approach
Defines exponential symmetries (continuous symmetries with linear generator A) and shows the Noether charge C = theta^T A theta drifts under SGD noise by a trace term; proves a unique attractive fixed point (noise equilibrium) exists along each degenerate manifold where noise contributions balance and align.
Key results
- Proves that for every exponential symmetry and every theta there exists a unique attractive fixed point (noise equilibrium) along the degenerate direction under SGD.
- For two-layer linear networks, SGD aligns W Gamma_W W^T with U^T Gamma_U U, while GD solutions remain far from this alignment (over an order of magnitude apart in experiments).
- For deep linear networks, the theory predicts balanced norms of all intermediate layers at the noise equilibrium, observed experimentally on MNIST.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adding SGD gradient noise (vs GD) in two-layer linear matrix factorization | two-layer linear network · autoencoding matrix factorization | alignment of learned layers (noise equilibrium condition) | Helped |
| Learning rate warmup before reaching large learning rate | SGD with Kaiming init on rank-1 matrix factorization · rank-1 matrix factorization | training stability (divergence) | Helped |
| Training with SGD (noise) rather than GD under scaling symmetry, with sharpness dynamics | Kaiming-initialized two-layer linear net · sharpness trace S during training | sharpness (trace of Hessian) | Helped |
| Data distribution choice affecting SGD flatness in matrix factorization | SGD on two-layer linear network · flatness of converged solution | sharpness | Mixed |
| Adding weight decay in place of SGD noise | two-layer linear network · norm balance of converged solution | norm balance | No clear effect |
| Removing weight decay under scale invariance (SGD noise only) | scale-invariant networks · parameter norm under SGD | parameter norm growth | Hurt |
Surprising findings
- SGD can drive sharpness either up or down depending on the data distribution, so SGD does not uniformly prefer flat minima.
- Without weight decay, parameter norms under SGD with scale invariance diverge monotonically, whereas GD keeps them constant.
Limitations
- Only covers problems describable by exponential symmetries; other symmetry types are not addressed.
- Theory relies on continuous-time SGF; discrete-time SGD holds only in expectation and practical reach of fixed points is easier with SGD than GD.
- Experiments are on small synthetic and toy settings (linear and two-layer nets, MNIST).
Experimental setup
Datasets: MNIST
Benchmarks: MNIST
Metrics: norm balance sharpness (trace of Hessian) alignment of W Sigma_x W and U Sigma_eps U
Baselines: gradient descent gradient descent with weight decay Xavier initialization Kaiming initialization
Builds on: stochastic gradient flow Noether's theorem Ito's lemma rescaling symmetry analysis of SGD
Closest papers by meaning
- Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics (ICLR 2021 · similarity 0.92)
- The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects (ICML 2019 · similarity 0.89)
- Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks (NeurIPS 2023 · similarity 0.89)
- Symmetries in Overparametrized Neural Networks: A Mean Field View (NeurIPS 2024 · similarity 0.89)
- On the hardness of learning under symmetries (ICLR 2024 · similarity 0.87)
- Symmetry Induces Structure and Constraint of Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.