Research map / Optimization / Implicit bias and generalization
On the Training Instability of Shuffling SGD with Batch Normalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use random reshuffling instead of single shuffle with batch normalization | linear + BN classifiers · classification; theoretical regimes and toy construction | GD-risk divergence | Helped |
| Use random reshuffling instead of single shuffle with batch normalization | deeper linear + BN networks · CIFAR10 | training-risk divergence | Helped |
| Use random reshuffling instead of single shuffle with batch normalization | 3-layer ReLU + BN MLP · CIFAR10, CIFAR100, MNIST | training-loss convergence speed | Helped |
| Use random reshuffling instead of single shuffle with batch normalization | finetuned ResNet18 · CIFAR10, CIFAR100, MNIST | training-loss convergence speed | Helped |
| Add batch normalization to single-shuffle SGD | linear networks · real datasets | training-risk divergence | Hurt |
| Use RR to average over the permutation-specific batch-normalization distortions | linear + BN regression · synthetic regression | distance of optimum from GD optimum | Helped |
| Tune the learning rate across the tested values (ablation) | SS + BN, deeper linear networks · CIFAR10 | training-risk divergence | No clear effect |
| Vary momentum across the tested values (ablation) | SS + BN, deeper linear networks · CIFAR10 | training-risk divergence | No clear effect |
| Vary batch size across the tested values (ablation) | SS + BN, deeper linear networks · CIFAR10 | training-risk divergence | No clear effect |
| Lower the learning rate (ablation) | SS + BN, finetuned ResNet18 · CIFAR10, CIFAR100, MNIST | training-loss convergence speed | No clear effect |
Closest papers by meaning
- Disentangling the Mechanisms Behind Implicit Regularization in SGD (ICLR 2023 · similarity 0.90)
- Stochastic Training is Not Necessary for Generalization (ICLR 2022 · similarity 0.89)
- A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent (ICML 2019 · similarity 0.88)
- Momentum Improves Normalized SGD (ICML 2020 · similarity 0.88)
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions (ICML 2025 · similarity 0.87)
- Tighter Convergence Bounds for Shuffled SGD via Primal-Dual Perspective (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.