Research map / Optimization / Implicit bias and generalization
Surprising Instabilities in Training Deep Networks and a Theoretical Analysis
Yuxin Sun, Dong Lao, Ganesh Sundaramoorthi, Anthony Yezzi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| perturb last floating-point bit of SGD update (k=3..11 vs k=1) | ResNet-56 SGD · CIFAR-10 | test accuracy STD: 0.16 | Hurt |
| replace ReLU with Swish activation (ablation) | perturbed SGD, ResNet-56 · CIFAR-10 | test accuracy variance: 93.72 ± 0.15 | No clear effect |
| use different network and dataset (VGG16, Fashion-MNIST) (ablation) | perturbed SGD · Fashion-MNIST | test accuracy variance | No clear effect |
| decrease fixed learning rate below a minimum (ablation) | perturbed SGD, ResNet-56 · CIFAR-10 weight divergence | relative L1 weight difference | Improved |
| learning rate dt < 0.03 in non-linear PDE (ablation) | one-layer CNN PDE · single image | error amplification | Improved |
| learning rate 0.03 < dt <= 0.1 in non-linear PDE (ablation) | one-layer CNN PDE · single image | error amplification | Hurt |
| learning rate dt > 0.1 in non-linear PDE (ablation) | one-layer CNN PDE · single image | loss | Hurt |
| weight decay inside (alpha_min, alpha_max) in linear PDE (ablation) | linearized gradient descent PDE · checkerboard input | loss convergence | Improved |
| finite fixed-support kernel window (16x16 to 64x64) vs infinite support (ablation) | linearized PDE · stability bounds on alpha | alpha_max: 2e8 | No clear effect |
| increase kernel size from 16x16 to 64x64 (ablation) | windowed linear PDE · stability bounds on alpha | empirical alpha_min: 3.8e6 -> 3.8e7 | Mixed |
| vary perturbation constant k (1,3,5,7,9,11) (ablation) | perturbed SGD · CIFAR-10 | final test accuracy: 92.99 to 93.87 | Mixed |
Closest papers by meaning
- A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks (JMLR 2024 · similarity 0.95)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.87)
- Train faster, generalize better: Stability of stochastic gradient descent (ICML 2016 · similarity 0.87)
- Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks (NeurIPS 2022 · similarity 0.86)
- Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods (ICLR 2025 · similarity 0.86)
- Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
