Research map / Architectures and efficiency / RNNs and normalization
On the spectral bias of two-layer linear networks
Aditya Vardhan Varre, Maria-Luiza Vladarean, Loucas Pillaud-Vivien, Nicolas Flammarion
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decrease initialization scale gamma from 1 to 10-4 (orthogonal init) (ablation) | two-layer linear network, gradient flow · synthetic vector regression (n=5, d=10, k=3, l=200) | rank of hidden layer W1: rank d -> approximately rank k=3 (gamma 1 -> 10−4) | Improved |
| use Gaussian initialization with variance 0.01 instead of orthogonal initialization (ablation) | two-layer linear network, gradient flow · synthetic vector regression (l=200, d=10) | rank of hidden layer W1 | No clear effect |
| add label noise to gradient descent (LNGD) (ablation) | two-layer linear network, larger init scale · synthetic scalar regression (n=d=5, l=200) | low-rank structure of hidden layer | Improved |
| decrease initialization scale to low scale in ReLU student network (ablation) | ReLU student network (20 neurons) with 2-neuron teacher · toy scalar regression, 10 samples in R5 | rank of hidden weight matrix | Improved |
Closest papers by meaning
- Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization (ICML 2026 · similarity 0.90)
- On the Explicit Role of Initialization on the Convergence and Implicit Bias of Overparametrized Linear Networks (ICML 2021 · similarity 0.90)
- On the Convergence of Gradient Flow on Multi-layer Linear Models (ICML 2023 · similarity 0.89)
- A unifying view on implicit bias in training linear neural networks (ICLR 2021 · similarity 0.88)
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.87)
- Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regression (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
