Research map / Learning theory and algorithms / Combinatorial and search optimization
Symmetry Induces Structure and Constraint of Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that every mirror-reflection symmetry in a loss function forces a linear stationary constraint O^T theta = 0, which weight decay or gradient noise favors, and applies this to rescaling, rotation, and permutation symmetries.
Problem
Understanding how symmetries in neural network loss functions shape the solutions and dynamics of gradient-based learning (sparsity, low rank, neuron homogeneity, collapse, loss of plasticity).
Approach
Unifies rescaling, rotation, and permutation symmetries as mirror (reflection) symmetries of the loss; shows each mirror induces a stationary condition O^T theta = 0 that weight decay and SGD gradient noise favor, and uses this to build a differentiable constraint-enforcing reparameterization (DCS).
Key results
- ResNet18 trained on CIFAR-10 with rescaling symmetry on each parameter (DCS/spred) reaches about 93% test accuracy, similar to vanilla, while the parameters are very sparse.
- Matrix factorization under SGD with large gradient noise converges to lower-rank solutions due to rotation symmetry; the residual-parameterized version (symmetry removed) stays full-rank.
- Loss-of-plasticity experiments show a linear regressor with rescaling symmetry accumulates dead neurons over 10 tasks, while vanilla regression does not; weak gradient noise or random bias fixes it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add rescaling symmetry by parameterizing weights as a Hadamard product | Linear regression · Noisy Gaussian data | sparsity | Helped |
| Add rescaling symmetry by parameterizing every parameter as a Hadamard product | ResNet18 · CIFAR-10 test | accuracy: roughly 93%; the DCS model higher by a small margin | Helped |
| Add rescaling symmetry by parameterizing every parameter as a Hadamard product | ResNet18 · CIFAR-10 | parameter sparsity | Helped |
| Add rescaling symmetry by parameterizing every parameter as a Hadamard product | ResNet18 · CIFAR-10 | training time | No clear effect |
| Add weight decay | ResNet18 · CIFAR-10 penultimate-layer activations | activation homogeneity | Helped |
| Add rotation symmetry by using vanilla matrix factorization rather than residual connections | Matrix factorization · 200-dimensional matrix factorization | rank | Helped |
| Remove rotation symmetry by introducing residual connections (ablation) | Matrix factorization · 200-dimensional matrix factorization | rank | Hurt |
| Add weight decay | Matrix factorization · Regularized matrix factorization | rank | Helped |
| Add weight decay | Transformer key and query weights · In-context learning | rank | Helped |
| Add rescaling symmetry by reparameterizing every weight as a Hadamard product | Linear regression · Continual learning across 10 tasks | dead neurons | Hurt |
| Add Gaussian noise to the gradient (ablation) | Rescaling-symmetric linear regression · Continual learning across 10 tasks | dead neurons | Helped |
| Add a fixed random bias to every weight parameter (ablation) | Rescaling-symmetric linear regression · Continual learning across 10 tasks | dead neurons | Helped |
Surprising findings
- Weight decay favors sparse or symmetric solutions in deep networks, unlike ordinary L2 regression which only becomes sparse at infinite weight decay.
- SGD noise makes symmetry-induced stationary conditions more attractive, the opposite of the naive expectation that noise helps escape them.
Limitations
- Theory relies on assumptions such as slowly-varying background dynamics and a Hessian eigenvalue lower bound for part 4 of the theorem.
- Experiments are small-scale (linear models, matrix factorization, tiny transformers, a single ResNet18 on CIFAR-10).
Experimental setup
Datasets: CIFAR-10 synthetic Gaussian regression data
Benchmarks: CIFAR-10
Metrics: sparsity fraction matrix rank neuron activation correlation block structure test accuracy dead neuron count
Baselines: vanilla linear regression vanilla matrix factorization vanilla ResNet18 spred
Builds on: spred lasso
Closest papers by meaning
- Remove Symmetries to Control Model Expressivity and Improve Optimization (ICLR 2025 · similarity 0.90)
- A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (NeurIPS 2025 · similarity 0.88)
- On the hardness of learning under symmetries (ICLR 2024 · similarity 0.88)
- Symmetries in Overparametrized Neural Networks: A Mean Field View (NeurIPS 2024 · similarity 0.88)
- Parameter symmetries determine representational geometry in overparameterized nonlinear networks (NeurIPS 2026 · similarity 0.88)
- Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.