Research map / Optimization / Implicit bias and generalization
Rethinking Gauss-Newton for learning over-parameterized models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use small hidden-weight initialization variance instead of large variance | Gauss-Newton and gradient descent · Synthetic teacher-student regression | test loss | Helped |
| Use large hidden-weight initialization variance | Gauss-Newton and gradient descent · Synthetic teacher-student regression | test loss | No clear effect |
| Use a smaller step size | Gauss-Newton · Synthetic teacher-student regression | test loss after linear refitting | Helped |
| Increase the step size | Gauss-Newton · Synthetic teacher-student regression | test loss after linear refitting | Hurt |
| Increase the step size | Gradient descent · Synthetic teacher-student regression | test loss after linear refitting | Helped |
| Use a larger step size | Gauss-Newton · Synthetic teacher-student regression | training loss convergence speed | Helped |
| Use a larger step size | Gauss-Newton · Synthetic teacher-student regression | generalization of learned features | Hurt |
| Use small step sizes so feature learning precedes linear-layer fitting | Gauss-Newton · Synthetic teacher-student regression | test loss after linear refitting | Helped |
| Replace ReLU with a SiLU activation (ablation) | Gauss-Newton and gradient descent · Synthetic teacher-student regression | transition between kernel and feature-learning regimes | No clear effect |
| Make SiLU nearly linear by decreasing its β parameter (ablation) | Gauss-Newton · Synthetic teacher-student regression | test performance relative to gradient descent | Hurt |
Closest papers by meaning
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.90)
- Theoretical Characterisation of the Gauss Newton Conditioning in Neural Networks (NeurIPS 2024 · similarity 0.90)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.90)
- Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure (ICLR 2025 · similarity 0.88)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.87)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.