Research map / Architectures and efficiency / RNNs and normalization
Learning dynamics of deep linear networks with multiple pathways
Jianghong Shi, Eric Todd SheaBrown, Michael A Buice
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase pathway depth from D = 2 to D = 7 (ablation) | two-pathway deep linear network · 8-dim identity input, 15-dim output simulation | concentration of each singular value in one pathway: D = 2 -> D = 7 | Improved |
| use depth 3 and 4 instead of depth 2 (ablation) | two-pathway deep linear network · 8-dim identity input, 15-dim output simulation | symmetry between pathways: D = 2 -> D = 3, 4 | Improved |
| use ReLU nonlinearity after each layer instead of linear (ablation) | two-pathway network · depths 2, 3, 4, 7 simulation | feature concentration in one pathway | No clear effect |
| use Tanh nonlinearity after each layer instead of linear (ablation) | two-pathway network · depths 2, 3, 4, 7 simulation | feature concentration in one pathway | No clear effect |
| larger initial weight fluctuations (sigma = 0.1 versus 0.01) (ablation) | two-pathway phase portrait, single alpha · S_alpha = 10 phase portrait | trajectory drift toward one dominant pathway: sigma = 0.01 -> sigma = 0.1 | Improved |
| increase hidden width to the infinite limit | multi-pathway linear network · analytic result | sharing of features across pathways | Improved |
| different random seed for initialization (ablation) | two-pathway network D = 7 · simulation, Na = Nb = 1000, sigma = 0.01 | which pathway holds the largest singular values | Mixed |
Closest papers by meaning
- Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons (NeurIPS 2026 · similarity 0.87)
- Brain-Like Processing Pathways Form in Models With Heterogeneous Experts (NeurIPS 2025 · similarity 0.87)
- How connectivity structure shapes rich and lazy learning in neural circuits (ICLR 2024 · similarity 0.86)
- Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · similarity 0.86)
- Critical Learning Periods Emerge Even in Deep Linear Networks (ICLR 2024 · similarity 0.86)
- Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
