Research map / Architectures and efficiency / RNNs and normalization
A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that parameter symmetries of an unconstrained MLP that are broken by equivariance constraints (hidden symmetries) can create critical points and spurious minima in permutation-equivariant networks, and that relaxing the constraint can let training escape to a global minimum in a differently-represented equivariant subspace.
Problem
Whether equivariance constraints themselves create fundamental optimization obstacles, rather than just requiring different hyperparameters, for equivariant networks built from permutation representations.
Approach
Views an equivariant network as a linearly constrained subspace of an unconstrained MLP; hidden parameter symmetries of the unconstrained model that the constraint breaks can have fixed-point subspaces intersecting the constrained space, inducing critical points and spurious minima via a symmetric-criticality argument.
Key results
- In a teacher-student S_3 setup with a pi_3 input block, the exact loss landscape shows two distinct minima, one of which is a hidden-symmetry-induced spurious minimum.
- Gradient descent (100 steps, lr 0.1) from bad initializations converges to the spurious minimum in one region of parameter space; the boundary aligns with the hidden fixed point subspace after rescaling the parameterization.
- Relaxing equivariance constraints from the spurious minimum lets the unconstrained network escape to a global minimum corresponding to a permuted, different hidden group representation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the S3-equivariant student from different initializations | equivariant student network · S3 teacher-student setup | final loss | Mixed |
| rescale the diagonal parameter as √(n−1)θ₁ instead of parameterizing diagonal and off-diagonal components directly (ablation) | equivariant student network · S3 teacher-student setup | shape of the convergence-basin boundary | Helped |
| relax the equivariance constraints after reaching a bad constrained minimum | equivariant student network · S3 teacher-student setup | loss | Helped |
| relax the equivariance constraints after reaching a bad constrained minimum | equivariant student network · S3 teacher-student setup | equivariance error | Mixed |
Surprising findings
- Relaxing equivariance can convert a spurious equivariant minimum into a saddle, letting the network jump to a different equivariant subspace with a different hidden representation.
- Parameterization/scaling choice changes the shape of the basin boundary, but some boundary (hidden fixed point subspace) persists regardless of rescaling.
Limitations
- Results require viewing the equivariant model as a subspace of a larger model class, which is hard to do for tensor-product-based equivariant models.
- Theory covers only fixed point subspaces and does not account for local parameter symmetries.
- Experiments are small toy teacher-student setups (2-layer ReLU, S_3) rather than practical equivariant architectures.
Experimental setup
Datasets: synthetic Gaussian inputs labeled by random teacher network
Metrics: final loss Hessian eigenvalues equivariance error
Baselines: equivariant student network (constrained training)
Builds on: principle of symmetric criticality permutation symmetry loss landscape analysis teacher-student ReLU minima analysis
Closest papers by meaning
- Learning (Approximately) Equivariant Networks via Constrained Optimization (NeurIPS 2025 · similarity 0.92)
- Improving Equivariant Networks with Probabilistic Symmetry Breaking (ICLR 2025 · similarity 0.92)
- Improving Equivariant Model Training via Constraint Relaxation (NeurIPS 2024 · similarity 0.92)
- A Characterization Theorem for Equivariant Networks with Point-wise Activations (ICLR 2024 · similarity 0.91)
- On the hardness of learning under symmetries (ICLR 2024 · similarity 0.91)
- Probing Equivariance and Symmetry Breaking in Convolutional Networks (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.