Research map / Optimization / Implicit bias and generalization
The Loss Surface Of Deep Linear Networks Viewed Through The Algebraic Geometry Lens
Dhagash B. Mehta, Tianran Chen, Tingting Tang, Jonathan D. Hauenstein
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add generalized L2 regularization to the deep linear loss (ablation) | deep linear network loss · stationary points | flat stationary points | Improved |
| increase input dimension d_x from 2 to 5 (H=1, m=2) | deep linear network, H=1 · isolated solutions | isolated complex solutions: 225 -> 225 | No clear effect |
| increase number of data points m from 2 to 20 (H=1) | deep linear network, H=1 · isolated solutions | isolated complex solutions: 225 -> 225 | No clear effect |
| increase number of hidden layers from H=1 to H=2 (m=1) | deep linear network · isolated solutions | isolated complex solutions: 33 -> 641 | Hurt |
| use BKK bound instead of classical Bezout bound (H=1, m=1, d_x=d_y=2) | bound on solutions · upper bound on solution count | bound size: 6561 -> 1024 | Improved |
| increase d_x from 2 to 10 (H=1, m=1) | deep linear network · BKK bound | BKK bound: 1024 -> 640000 | Hurt |
| increase d_y from 2 to 5 (H=1, m=1) | deep linear network · complex solutions | isolated complex solutions: 33 -> 201 | Hurt |
| increase range of regularization Lambda beyond 1 | regularized deep linear network · real solutions | mean number of real solutions | Hurt |
| increase number of data points m | regularized deep linear network · real solutions | mean number of real solutions | Improved |
| use nonzero regularization with H=1, m=5, d_x=d_y=3 | regularized deep linear network · local minima | local minima that are global: 7.16775 vs 7.13717 | Hurt |
Closest papers by meaning
- Unique Properties of Flat Minima in Deep Networks (ICML 2020 · similarity 0.90)
- Geometry of Neural Network Loss Surfaces via Random Matrix Theory (ICML 2017 · similarity 0.89)
- The Loss Surface of Deep and Wide Neural Networks (ICML 2017 · similarity 0.89)
- Topology and Geometry of Half-Rectified Network Optimization (ICLR 2017 · similarity 0.89)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.87)
- Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
