Research map / Optimization / Implicit bias and generalization
Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data
Dachao Lin, Ruoyu Sun, Zhihua Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use spherically symmetric data instead of a finite positive-margin dataset for logistic regression directional convergence | gradient descent on logistic regression · directional convergence rate | convergence rate of weight direction | Improved |
| increase depth from N=1 or 2 to N>=3 layers in the linear network | deep linear network under gradient flow · spherically symmetric data | effect of weight norm on directional convergence speed | Mixed |
| scale up the data to increase expected norm c0 | second-phase convergence of deep linear network · spherically symmetric data | second-phase convergence speed | Improved |
| add the second phase (weight norm ascending) to training dynamics | deep linear network gradient flow · x ~ U(S1) simulation | directional convergence rate | Improved |
| run SGD with batch size 1000 and learning rate 10^-3 instead of exact gradient flow | linear classifier and 4-layer linear network · x ~ U(S1) simulation | match to theoretical bounds | No clear effect |
| track accuracy-related direction cos(theta) instead of loss alone | deep linear network training analysis · x ~ U(S1) simulation | progress during early training | Improved |
Closest papers by meaning
- Directional convergence and alignment in deep learning (NeurIPS 2020 · similarity 0.89)
- A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks (ICLR 2019 · similarity 0.88)
- Gradient descent aligns the layers of deep linear networks (ICLR 2019 · similarity 0.87)
- Convergence beyond the over-parameterized regime using Rayleigh quotients (NeurIPS 2022 · similarity 0.87)
- Global Convergence of Gradient Descent for Deep Linear Residual Networks (NeurIPS 2019 · similarity 0.87)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
