Research map / Architectures and efficiency / RNNs and normalization
Remove Symmetries to Control Model Expressivity and Improve Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that loss symmetries cause low-capacity collapses and proposes syre, adding a random static bias to parameters combined with weight decay, to remove reflection symmetries without knowing them.
Problem
Symmetries in neural network loss functions create low-capacity saddle points (collapses) that trap training and ignore features, and existing methods cannot fully escape them, especially unknown or hidden symmetries.
Approach
Shift parameters by a small random static bias theta_0 inside the weight decay term (loss l(theta+theta_0) + gamma||theta||^2), which provably removes all discrete reflection symmetries with probability 1 without knowledge of the symmetry; extended with a random diagonal metric D for uncountably many symmetries.
Key results
- On ResNet-18 with CIFAR-10, syre matches vanilla accuracy for sigma_0 < 0.2 (no significant difference).
- In linear probing of SimCLR ResNet-18 on CIFAR-100, syre raises last-layer accuracy from 22.2% to 32.5% (sigma_0=0.01), versus 46.8% penultimate layer.
- In the controlled objective benchmark, syre is the only method that smoothly interpolates between low symmetry and optimized objective values; syre adds negligible time and memory overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply syre to the projection head | ResNet18 SimCLR representation · CIFAR-100, linear evaluation, last layer | last-layer accuracy: 22.2% -> 31.7% | Helped |
| Apply syre to the projection head | ResNet18 SimCLR representation · CIFAR-100, linear evaluation, penultimate layer | low-rankness: 70% -> 0% | Helped |
| Apply syre to the projection head | ResNet18 SimCLR representation · CIFAR-100, linear evaluation, penultimate layer | penultimate-layer accuracy: 46.8% -> 46.8% | No clear effect |
| Use σ₀=0.01 instead of σ₀=0.1 while applying syre to the projection head (ablation) | ResNet18 SimCLR representation · CIFAR-100, linear evaluation, last layer | last-layer accuracy: 31.7% -> 32.5% | Helped |
| Apply syre to all layers rather than only the projection head (ablation) | ResNet18 SimCLR representation with σ₀=0.1 · CIFAR-100, linear evaluation, penultimate layer | penultimate-layer accuracy: 46.8% -> 44.6% | Hurt |
| Apply syre to all layers rather than only the projection head (ablation) | ResNet18 SimCLR representation with σ₀=0.1 · CIFAR-100, linear evaluation, last layer | last-layer accuracy: 31.7% -> 30.7% | Hurt |
| Use σ₀=0.01 instead of σ₀=0.1 and apply syre to all layers (ablation) | ResNet18 SimCLR representation · CIFAR-100, linear evaluation, penultimate layer | penultimate-layer accuracy: 44.6% -> 45.4% | Helped |
| Use σ₀=0.01 instead of σ₀=0.1 and apply syre to all layers (ablation) | ResNet18 SimCLR representation · CIFAR-100, linear evaluation, last layer | last-layer accuracy: 30.7% -> 32.4% | Helped |
Surprising findings
- Adding a random static bias does not change performance under standard training, yet removes rank collapse in the representation.
- Symmetry removal alone does not close the gap between last-layer and penultimate-layer representations in SimCLR, suggesting symmetry is not the only cause.
Limitations
- Theoretical guarantees are stated for reflection symmetries and finite/discrete groups; the paper's removal strength can scale inversely with number of symmetries (1/N or 1/log|U|).
- The choice of sigma_0 is a hyperparameter that trades off optimization against symmetry removal; a default of 0.01/sqrt(d) is heuristic.
- Experiments are mostly on small or medium-scale models; deployment to large language models is left as future work.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 Fashion-MNIST Permuted MNIST Slippery-Ant (PyBullet Ant variant)
Benchmarks: CIFAR-10 CIFAR-100 MNIST Fashion-MNIST Slippery-Ant
Metrics: test accuracy training loss effective rank of covariance reconstruction loss returns degree of symmetry
Baselines: vanilla training (SGD/Adam) weight decay W-fix dropout W-asymmetric networks sigma-asymmetric networks
Builds on: weight decay reflection symmetry theory lazy training (NTK) beta-VAE PPO
Closest papers by meaning
- Symmetry Induces Structure and Constraint of Learning (ICML 2024 · similarity 0.90)
- The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof (NeurIPS 2024 · similarity 0.86)
- Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent (NeurIPS 2024 · similarity 0.86)
- Symmetries in Overparametrized Neural Networks: A Mean Field View (NeurIPS 2024 · similarity 0.86)
- A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (NeurIPS 2025 · similarity 0.86)
- On the hardness of learning under symmetries (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.