Research map / Optimization / Implicit bias and generalization
A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks
Yuxin Sun, Dong Lao, Anthony Yezzi, Ganesh Sundaramoorthi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| perturb SGD update at the last floating point bit (k=3 to 11) (ablation) | SGD · ResNet-56 CIFAR-10 ReLU | final test accuracy STD: 0.11 -> 0.22 | Hurt |
| increase precision from 32 bit to 64 bit | SGD · ResNet-56 CIFAR-10 | test accuracy STD: 0.16 -> 0.15 | No clear effect |
| replace ReLU with Swish activation (ablation) | perturbed SGD · ResNet-56 CIFAR-10 | test accuracy STD under float perturbation: 0.16 -> 0.15 | No clear effect |
| switch SGD to Adam optimizer (ablation) | perturbed SGD · ResNet-56 CIFAR-10 Swish | test accuracy STD: 0.22 -> 0.28 | No clear effect |
| use VGG16BN instead of ResNet56 and Fashion-MNIST instead of CIFAR-10 (ablation) | perturbed SGD · Swish | test accuracy STD: 94.81 ± 0.09 | No clear effect |
| multiply k by a power of two (k=1 vs k=2,4) (ablation) | perturbed SGD · ResNet-56 CIFAR-10 | final test accuracy: 93.47 -> 93.47 | No clear effect |
| decrease learning rate to a stable value 3.125e-6 (ablation) | SGD · ResNet-56 CIFAR-10 | weight divergence between k=1 and k=3 | Improved |
| use 3-layer CNN instead of 1-layer CNN at learning rate 5 (ablation) | gradient descent · single-datum CNN | loss stability | Hurt |
| double the layers of a 2-layer FC network without weight decay (ablation) | gradient descent · CIFAR-5k | loss stability | Hurt |
| increase learning rate from 0.03 to 0.05 (ablation) | gradient descent · 1-layer CNN single datum | normalized sharpness | Hurt |
Closest papers by meaning
- Surprising Instabilities in Training Deep Networks and a Theoretical Analysis (NeurIPS 2022 · similarity 0.95)
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (ICLR 2021 · similarity 0.90)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.89)
- Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods (ICLR 2025 · similarity 0.88)
- A Minimalist Example of Edge-of-Stability and Progressive Sharpening (NeurIPS 2025 · similarity 0.88)
- Variational Learning Finds Flatter Solutions at the Edge of Stability (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
