Research map / Optimization / Implicit bias and generalization
Stochastic Training is Not Necessary for Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace mini-batch SGD with full-batch training using clipping, gradient regularization, and accumulation batch size 32 | ResNet-18 · CIFAR-10 validation with randomized augmentation | accuracy (%): 95.70 (± 0.11) -> 95.68 (± 0.09) | No clear effect |
| Train full-batch for 3000 steps instead of 300 (ablation) | Full-batch ResNet-18 · CIFAR-10 validation with randomized augmentation | accuracy (%): 75.42 (± 0.13) -> 87.36 (± 1.23) | Helped |
| Add full-dataset gradient clipping (ablation) | Full-batch ResNet-18 trained longer · CIFAR-10 validation with randomized augmentation | accuracy (%): 87.36 (± 1.23) -> 93.85 (± 0.10) | Helped |
| Add gradient regularization (ablation) | Clipped full-batch ResNet-18 · CIFAR-10 validation with randomized augmentation | accuracy (%): 93.85 (± 0.10) -> 95.54 (± 0.09) | Helped |
| Reduce gradient-regularization accumulation batch size from 128 to 32 (ablation) | Regularized full-batch ResNet-18 · CIFAR-10 validation with randomized augmentation | accuracy (%): 95.54 (± 0.09) -> 95.68 (± 0.09) | Helped |
| Increase learning rate from 0.4 to 0.8 (ablation) | Clipped full-batch ResNet-18 without gradient regularization · CIFAR-10 validation with randomized augmentation | accuracy (%): 93.85 (± 0.10) -> 93.75% (± 0.13) | Hurt |
| Increase learning rate from 0.4 to 0.8 while using gradient regularization (ablation) | Regularized full-batch ResNet-18 · CIFAR-10 validation with randomized augmentation | accuracy (%): 94.70% (± 0.17) -> 95.54 (± 0.09) | Helped |
| Remove gradient clipping (ablation) | FB strong reg. ResNet-18 · CIFAR-10 validation with randomized augmentation | accuracy (%): 95.68 (± 0.09) -> 91.08% (-) | Hurt |
| Add gradient regularization to SGD (ablation) | SGD ResNet-18 · CIFAR-10 validation with randomized augmentation | accuracy (%): 95.70 (± 0.11) -> 95.81% (± 0.18) | No clear effect |
| Remove randomized data augmentation and use clipped, regularized full-batch training | ResNet-18 · CIFAR-10 validation without random augmentation | accuracy (%): 84.32 (± 1.12) -> 89.17 (± 0.24); tuned SGD: 90.07 (± 0.48) | Mixed |
| Expand the training dataset 10× with fixed augmented samples and train full-batch with clipping and regularization | ResNet-18 · 10× CIFAR-10 validation, fixed augmented dataset | accuracy (%): 95.20 (± 0.09) -> 95.11(-) | No clear effect |
| Use data shuffling with the full-batch method (ablation) | FB strong reg. ResNet-18 · CIFAR-10 validation with randomized augmentation | accuracy (%): 95.68 (± 0.09) -> 95.91 (± 0.14) | No clear effect |
Closest papers by meaning
- Disentangling the Mechanisms Behind Implicit Regularization in SGD (ICLR 2023 · similarity 0.93)
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (ICLR 2017 · similarity 0.91)
- Don't Use Large Mini-batches, Use Local SGD (ICLR 2020 · similarity 0.91)
- SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs (NeurIPS 2021 · similarity 0.90)
- Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions (NeurIPS 2022 · similarity 0.90)
- SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.