Research map / Optimization / Implicit bias and generalization
Bad Global Minima Exist and SGD Can Reach Them
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain on random labels before training on true labels | vanilla SGD, ResNet18 · CIFAR10 | test accuracy (%): 84.838 -> 56.024 | Hurt |
| pretrain on random labels before training on true labels | SOTA SGD (DA+l2+momentum), ResNet18 · CIFAR10 | test accuracy (%): 95.324 -> 95.346 | No clear effect |
| add data augmentation to vanilla SGD (adversarial init) (ablation) | vanilla SGD, ResNet18 · CIFAR10 | test accuracy (%): 56.024 -> 89.402 | Helped |
| add l2 regularization to vanilla SGD (adversarial init) (ablation) | vanilla SGD, ResNet18 · CIFAR10 | test accuracy (%): 56.024 -> 83.172 | Helped |
| add momentum to vanilla SGD (adversarial init) (ablation) | vanilla SGD, ResNet18 · CIFAR10 | test accuracy (%): 56.024 -> 89.016 | Helped |
| add a second heuristic on top of data augmentation (l2) (ablation) | SGD with DA, adversarial init · CIFAR10 ResNet18 | test accuracy (%): 89.402 -> 94.192 | Helped |
| increase replication factor R of random-label pretraining (ablation) | vanilla SGD · CIFAR10 | test accuracy | Hurt |
Closest papers by meaning
- Label Noise SGD Provably Prefers Flat Global Minimizers (NeurIPS 2021 · similarity 0.91)
- Disentangling the Mechanisms Behind Implicit Regularization in SGD (ICLR 2023 · similarity 0.91)
- SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs (NeurIPS 2021 · similarity 0.90)
- The Benefits of Implicit Regularization from SGD in Least Squares Problems (NeurIPS 2021 · similarity 0.89)
- Local Regularizer Improves Generalization (AAAI 2020 · similarity 0.89)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.