Research map / Learning theory and algorithms / Prediction and decision losses
Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analytically characterises the solution manifold of two-layer deep linear networks and shows that functional equivalence and representational similarity are dissociable, with only parameter-noise robustness favouring task-specific representations.
Problem
Representational similarity analysis and linear predictivity are used to infer function from neural codes, yet it is unclear how a network's hidden representations relate to its input-output function when many parametrisations are functionally equivalent.
Approach
Derives exact parametric equations for the global-minimum solution manifold of two-layer linear networks, partitions it into subregions (GLS, LSS, MRNS, MWNS) with task-agnostic vs task-specific representational similarity, and analyses how each relates to secondary error, input noise, and parameter noise robustness.
Key results
- Parameter-noise robustness is minimised exclusively by MRNS (theory), while input-noise robustness is minimised exclusively by LSS, which includes MWNS as a subspace.
- In linear predictivity, across-function R^2 is highest when the source is an LSS (task-agnostic) and lowest for MWNS, so predictivity is driven mainly by solution type rather than functional alignment.
- In nonlinear ReLU networks on MNIST, expanded non-minimal parametrisations (scaled, nuisance, duplicate) degrade faster under parameter noise, and input-null expansions degrade under input noise, while none inflate test error at near-zero noise.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Allow arbitrary globally optimal weight parametrisations | Two-layer linear networks · Global minima and hidden representations | representational flexibility | Helped |
| Use minimum representation-and-readout norm solutions (MRNS) | Two-layer linear networks · Hidden representations | RSM uniqueness | Helped |
| Use minimum weight-norm solutions (MWNS) | Two-layer linear networks · Hidden representations | RSM uniqueness | Helped |
| Compare representations from within-function rather than across-function solutions | Linear predictivity comparisons · Random regression task | linear predictivity R² | Helped |
| Use a task-agnostic LSS representation as the source | Linear predictivity comparisons · Random regression task | linear predictivity R² | Helped |
| Predict a task-agnostic representation from a task-specific one | Linear predictivity comparisons · Random regression task | linear predictivity R² | Hurt |
| Use task-agnostic solutions with flexible RSMs | Representational similarity analysis · Random walks on linear solution manifolds | RSA correlation stability | Hurt |
| Walk on a functionally equivalent solution manifold | Linear network representations · Random regression task | linear decoder performance | Hurt |
| Select solutions for robustness to input noise | Two-layer linear networks · Input noise | expected loss | Helped |
| Select solutions for robustness to input noise | Two-layer linear networks · Hidden representations | task-specific representation | No clear effect |
| Select solutions for robustness to parameter noise | Two-layer linear networks · Parameter noise | expected loss | Helped |
| Add input-nullspace expansion | Task-trained ReLU network · MNIST test set, input noise | test error | Hurt |
Surprising findings
- Linear predictivity and RSA can be dominated by solution type rather than by whether two networks implement the same function.
- Generalisation (secondary) error does not favour task-specific representations, since minimising it permits task-agnostic solutions.
Limitations
- Main theory is restricted to two-layer linear networks; nonlinear results are numerical and do not fully characterise the nonlinear solution manifold.
- Input-noise and parameter-noise results rely on whitened inputs or scaled noise variances for clean expressions.
- The paper does not analyse which solution type gradient descent converges to.
Experimental setup
Datasets: MNIST synthetic random regression semantic hierarchy task
Benchmarks: MNIST
Metrics: R^2 linear predictivity RSA correlation MSE of linear decoder test error under input noise test error under parameter noise
Baselines: linear predictivity representational similarity analysis minimum-norm solutions
Builds on: Laurent and Brecht 2018 deep linear network no bad local minima Saxe et al. 2014 exact solutions Martinelli et al. 2023 expand-and-cluster Simsek et al. 2021 geometry of loss landscape
Closest papers by meaning
- Parameter symmetries determine representational geometry in overparameterized nonlinear networks (NeurIPS 2026 · similarity 0.89)
- Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.89)
- Formation of Representations in Neural Networks (ICLR 2025 · similarity 0.88)
- Task structure and nonlinearity jointly determine learned representational geometry (ICLR 2024 · similarity 0.87)
- How are linear representations learned? Exact solutions to the dynamics of abstraction (NeurIPS 2026 · similarity 0.87)
- Neural networks are more modular than single neurons suggest (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.