Research map / Optimization / Implicit bias and generalization
No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically measures the restricted secant inequality (RSI), error bound (EB) and their ratio (cosine similarity between sampled gradients and the direction to the final iterate) along neural network training paths, finding the cosine stays positive and stable and using it to motivate learning rate schedules.
Problem
Explaining why stochastic first-order methods efficiently train neural networks despite non-convex loss landscapes, by characterizing the local geometry of sampled gradients along optimization paths without assuming smoothness.
Approach
Measures local RSI and EB quantities (and their ratio, a gradient-to-final-iterate cosine similarity) on sampled minibatch gradients along training trajectories, using a two-run protocol where the first run yields the final iterate as reference point, and assumes differentiability but not smoothness.
Key results
- On CIFAR-10 with ResNet-18, gamma never becomes negative across the entire run and mostly stays within [0.0075, 0.02].
- On WikiText-2 with a Transformer, gamma stays strictly positive and exceeds 0.05 after the second epoch.
- On ImageNet with ResNet-50, gamma is negative only in rare iterations and exceeds 0.0018 for most of training; in contrast, synthetic counterexamples (ALM, SM) yield negative RSI and gamma.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase batch size (ablation) | Transformer · WikiText-2 | cosine similarity γ | Helped |
| increase batch size (ablation) | ResNet-18 · CIFAR-10 | cosine similarity γ | Helped |
| use Adam instead of SGD (ablation) | ResNet-18 · CIFAR-10 | cosine similarity γ | Hurt |
| add momentum (β=0.9) to SGD (ablation) | ResNet-18 · CIFAR-10 | cosine similarity γ | No clear effect |
| increase ResNet depth (ablation) | ResNet architectures · ImageNet | cosine similarity γ | Helped |
| increase ResNet width (ablation) | ResNet architectures · ImageNet | cosine similarity γ | No clear effect |
| vary initialization seed (ablation) | ResNet-18 · CIFAR-10 | cosine similarity γ | No clear effect |
| vary epoch budget from 100 to 280 (ablation) | ResNet-18 · CIFAR-10 | cosine similarity γ: 100 to 280 | No clear effect |
| use a wide 4-layer MLP instead of a double-width ResNet-18 (ablation) | CIFAR-10 classifier · CIFAR-10 | cosine similarity γ regularity | Helped |
| train on ImageNet-1K with ResNet-50 | neural network optimization trajectories · ImageNet-1K | cosine similarity γ positivity: 0.0018 | Helped |
| train on WikiText-2 with a Transformer | neural network optimization trajectories · WikiText-2 | cosine similarity γ positivity: 0.05 | Helped |
| train on CIFAR-10 with ResNet-18 | neural network optimization trajectories · CIFAR-10 | cosine similarity γ: [0.0075, 0.02] | Helped |
Surprising findings
- An MLP showed greater cosine-similarity regularity than ResNet-18 despite worse performance, suggesting the geometry is not solely due to architectural tuning.
- Adam yields lower cosine similarity than SGD variants, though with reduced variance.
Limitations
- Reference point w* is the final iterate of the same optimization sequence, which induces a correlation bias that inflates gamma near the end of training.
- Measured loLR is not globally optimal and cannot be used to tune the learning rate dynamically since it depends on the learning rate itself.
- Late-training measurements suffer precision issues and are dominated by the interpolation/non-interpolation regime effects.
Experimental setup
Datasets: CIFAR-10 ImageNet-1K WikiText-2 Vaihingen
Benchmarks: CIFAR-10 ImageNet-1K WikiText-2 Vaihingen
Metrics: cosine similarity gamma RSI EB validation accuracy validation perplexity
Baselines: SGD SGD with momentum Adam
Builds on: restricted secant inequality error bound gradient descent optimality under RSI and EB
Closest papers by meaning
- The Directionality of Optimization Trajectories in Neural Networks (ICLR 2025 · similarity 0.90)
- Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization (ICLR 2020 · similarity 0.89)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.89)
- Loss Landscape Characterization of Neural Networks without Over-Parametrization (NeurIPS 2024 · similarity 0.89)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.88)
- Topological Generalization Bounds for Discrete-Time Stochastic Optimization Algorithms (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.