Research map / Optimization / Implicit bias and generalization
ON THE ROLE OF IMPLICIT REGULARIZATION OF STOCHASTIC GRADIENT DESCENT IN GROUP ROBUSTNESS
Nahal Mirzaie, Alireza Alipanah, Ali Abbasi, Amirmahdi Farzane, Hossein Jafarinia, Erfan Sobhaei, Mahdi Ghaznavi, Amir Najafi, Mahdieh Soleymani Baghshah, Mohammad Hossein Rohban
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reduce batch size from 256 to 8 (ablation) | SGD · CMNIST (rho=5%) | worst-group accuracy: 64.7 -> 67.5 | Improved |
| reduce batch size from 256 to 8 (ablation) | SGD · Domino (rho=5%) | worst-group accuracy: 43.4 -> 59.3 | Improved |
| reduce batch size from 256 to 8 (ablation) | SGD · Waterbirds (rho=5%) | worst-group accuracy: 68.2 -> 79.7 | Improved |
| reduce batch size from 256 to 8 (ablation) | SGD · CelebA (rho=5%) | worst-group accuracy: 45.0 -> 46.0 | Mixed |
| reduce batch size from 256 to 8 (ablation) | SGD · CIFAR10 (rho=5%) | worst-group accuracy: 77.9 -> 80.1 | Improved |
| reduce batch size from 256 to 8 (ablation) | SGD with BERT · Multi-NLI | worst-group accuracy: 75.17 -> 76.75 | Improved |
| reduce batch size from 256 to 8 (ablation) | SGD with BERT · Multi-NLI | accuracy: 79.34 -> 82.18 | Improved |
| reduce batch size from 128 to 8 (ablation) | SGD with BERT · CivilComments | worst-group accuracy: 53.70 -> 60.72 | Improved |
| increase learning rate while accuracy is near optimal (ablation) | SGD · five spurious-correlation benchmarks | worst-group accuracy: WGA increases with learning rate, reaches a near-optimal point, then declines | Mixed |
| use small batch size when training explicit debiasing method (ablation) | DFR · Domino-CMF | worst-group accuracy: 42.7 -> 72.9 | Improved |
| use small batch size when training explicit debiasing method (ablation) | AFR · Domino-CMF | worst-group accuracy: 40.3 -> 67.3 | Improved |
| use small batch size when training explicit debiasing method (ablation) | EVaLS · Domino-CMF | worst-group accuracy: 51.2 -> 76.7 | Improved |
Closest papers by meaning
- Disentangling the Mechanisms Behind Implicit Regularization in SGD (ICLR 2023 · similarity 0.92)
- The Benefits of Implicit Regularization from SGD in Least Squares Problems (NeurIPS 2021 · similarity 0.90)
- The Price of Implicit Bias in Adversarially Robust Generalization (NeurIPS 2024 · similarity 0.89)
- Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks (ICLR 2025 · similarity 0.88)
- Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning (ICML 2024 · similarity 0.88)
- The Implicit Bias of Gradient Descent toward Collaboration between Layers: A Dynamic Analysis of Multilayer Perceptions (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
