Research map / Optimization / Implicit bias and generalization
Explicit loss asymptotics in the gradient descent training of neural networks
Maksim Velikanov, Dmitry Yarotsky
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase network depth L from 1 to 3 hidden layers | NTK-regime ReLU network with GP target, d=3 · Synthetic GP target, d=3 | loss exponent xi and eigenvalue exponent nu | No clear effect |
| Vary activation smoothness q in (z)_+^q (0.5, 0.75, 1.0, 1.5) | Shallow NTK network · NTK eigenvalues, d=1 and d=3 | eigenvalue exponent nu | Mixed |
| Train in the mean-field regime instead of the NTK regime | Shallow ReLU network · GP target, d=4 | eigenvalue exponent nu | No clear effect |
| Use only the singular part of the NTK instead of the full NTK (ablation) | NTK eigenvalue spectrum · d = 1, 3, 6, 10; M = 100, 1000, 10000 | eigenvalues lambda_n | No clear effect |
| Increase input dimension d | NTK eigenvalue convergence to asymptotic law · d = 1, 3, 6, 10 | index n at which spectrum matches asymptotic | Hurt |
| Switch target from GP-generated to indicator of a ball | Shallow ReLU NTK network · d=2, d=4 | loss exponent xi: 3/(d+1) -> 1/(d+1) | Hurt |
| Apply theory to real data (MNIST) | Relation xi = kappa/nu · MNIST | fitted exponents: 0.26 ≈ 0.33/1.35 | Improved |
Closest papers by meaning
- SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures (NeurIPS 2025 · similarity 0.87)
- Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization (NeurIPS 2024 · similarity 0.87)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.86)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.86)
- Asymmetric Scaling Laws from Sparse Features (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
