Research map / Architectures and efficiency / RNNs and normalization
Parameter symmetries determine representational geometry in overparameterized nonlinear networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that function-preserving parameter symmetries in one-hidden-layer nonlinear networks act on hidden representations via three primitive feature transformations (addition, duplication, scaling), decomposes RSMs into essential and auxiliary parts, and proves minimum-weight-norm selection can yield identifiable representational geometry.
Problem
Representational geometry (RSM) is often used to infer computation, but parameter symmetries let networks realize the same function with very different hidden representations; it is unclear how much function constrains representation in nonlinear networks.
Approach
Characterizes parameter symmetries of nonlinear one-hidden-layer networks at the feature level as compositions of addition, duplication, and scaling; uses this to split the RSM into essential and auxiliary rank-one components, show similarity spread grows with width until saturation, and restore identifiability via minimum-weight-norm selection.
Key results
- Six distinct two-neuron ReLU XOR solutions with same function can have nearly uncorrelated RSMs, while different-function networks can have highly correlated RSMs (double dissociation).
- Similarity spread across widths grows strictly until saturation; for rank-full probe inputs correlations arbitrarily close to 1 and -1 are attainable in every orbit.
- Under minimum-weight-norm selection with linearly independent class residual matrices, all minimum-norm parameterizations share a unique RSM independent of width.
Surprising findings
- Same function can be realized with nearly uncorrelated representational geometries, and the reachable similarity range is function-independent in the limit of large width.
- Overparameterization can make the RSM arbitrarily aligned with a reference geometry of a different function, with auxiliary contributions dominating.
Limitations
- Theoretical; results are for one-hidden-layer networks and do not address cross-layer degeneracy.
- Results characterize symmetry-generated variation and do not claim to exhaust the full fiber of a realized function.
- Identifiability proof relies on minimum-norm selection and linear independence conditions.
Experimental setup
Datasets: XOR task (two-neuron ReLU networks from gradient descent)
Metrics: Pearson correlation between RSMs spread of attainable similarity scores
Baselines: Braun et al. 2025 linear-network results
Builds on: Simsek et al. 2021 geometry of the loss landscape (expansion manifold) Martinelli et al. 2024 expand-cluster symmetries Braun et al. 2025 dissociation of function and representation in linear networks Neyshabur et al. 2015 path-SGD
Closest papers by meaning
- The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof (NeurIPS 2024 · similarity 0.89)
- Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks (ICML 2025 · similarity 0.89)
- Task structure and nonlinearity jointly determine learned representational geometry (ICLR 2024 · similarity 0.89)
- Symmetries in Overparametrized Neural Networks: A Mean Field View (NeurIPS 2024 · similarity 0.88)
- Formation of Representations in Neural Networks (ICLR 2025 · similarity 0.88)
- A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.