Research map / Optimization / Optimizers
Loss Landscape Characterization of Neural Networks without Over-Parametrization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the alpha-beta-condition, a function class that captures non-convex loss landscapes with saddle points and local minima without heavy over-parametrization, proves SGD-type convergence under it, and validates it empirically across many architectures.
Problem
Existing convergence conditions (PL, Aiming, QCvx) for gradient-based optimizers either require impractically large over-parametrization for neural networks or exclude saddle points and local minima, so they do not describe realistic deep network loss landscapes.
Approach
Defines the alpha-beta-condition on individual losses f_i relative to the projection onto the minimizer set, which allows saddle points and local minima; it generalizes QCvx and yields non-vanishing-neighborhood convergence for SGD, SPS_max and NGN.
Key results
- Proves SGD with constant stepsize gamma <= (alpha-beta)/(2L) converges at rate O(K^{-1/2}) up to a neighborhood of size O(beta sigma_int^2).
- Empirically the alpha-beta-condition holds along training trajectories for MLP, CNN, ResNet9/18/34, LSTM, AlgoPerf workloads and Pythia models, with the Aiming and PL conditions failing in the LSTM and half-space settings.
- Shows that Adam and SGDM explore landscapes with smaller alpha and beta values than plain SGD on ResNet9 (CIFAR100).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the width of the 3-layer MLP | MLP · Fashion-MNIST | minimum feasible α and β | Mixed |
| Increase the width of the 3-layer MLP | SGD convergence neighborhood · Fashion-MNIST | convergence neighborhood size | Helped |
| Vary the number of convolutions in the second layer | CNN · CIFAR10 | minimum feasible α and β | Mixed |
| Increase batch size | ResNet9 · CIFAR100 | minimum feasible α and β | Helped |
| Increase batch size | ResNet9 · CIFAR100 | α−β | Helped |
| Use SGDM instead of SGD | ResNet9 · CIFAR100 | α and β | Helped |
| Use Adam instead of SGDM | ResNet9 · CIFAR100 | α and β | Helped |
| Increase ResNet depth | ResNet · CIFAR100 | α and β | Helped |
Surprising findings
- The Aiming and PL conditions are violated in practical LSTM training, with PL constant needing to be as small as 1e-8 to 1e-7.
- The alpha-beta-condition values for large AlgoPerf workloads are higher than for smaller models, attributed to less interpolation.
Limitations
- Empirical verification only along trajectories of specific optimizers and seeds, not the full loss landscape.
- Theoretical alpha and beta values for neural networks are not tight or precisely computed.
- Convergence guarantees are only up to a neighborhood whose size is governed by beta times interpolation error.
Experimental setup
Datasets: Fashion-MNIST CIFAR10 CIFAR100 Wikitext-2 Penn Treebank Criteo 1TB FastMRI OGBG WMT SlimPajama
Benchmarks: Fashion-MNIST CIFAR10 CIFAR100 Wikitext-2 Penn Treebank AlgoPerf workloads Pythia pretraining on SlimPajama
Metrics: minimum feasible (alpha, beta) pairs satisfying the conditio convergence bounds on function suboptimality
Baselines: PL condition Aiming condition QCvx SGDM Adam
Builds on: quasar-convexity Polyak-Lojasiewicz condition Aiming condition Stochastic Polyak Stepsize NGN (Non-negative Gauss-Newton)
Closest papers by meaning
- Large Scale Structure of Neural Network Loss Landscapes (NeurIPS 2019 · similarity 0.91)
- Convergence Rates of Non-Convex Stochastic Gradient Descent Under a Generic Lojasiewicz Condition and Local Smoothness (ICML 2022 · similarity 0.90)
- Convergence beyond the over-parameterized regime using Rayleigh quotients (NeurIPS 2022 · similarity 0.89)
- Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel (NeurIPS 2025 · similarity 0.89)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.89)
- Improving Adaptivity via Over-Parameterization in Sequence Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.