Research map / Optimization / Implicit bias and generalization
Disentangling the Mechanisms Behind Implicit Regularization in SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add average micro-batch Fisher trace regularization | LB SGD, ResNet-18 · CIFAR100, Tiny-ImageNet, SVHN | test accuracy | Helped |
| Add average micro-batch gradient norm regularization | LB SGD, ResNet-18 · CIFAR10, CIFAR100, Tiny-ImageNet, SVHN | test accuracy | Helped |
| Replace loss-output-gradient weighting with average Jacobian regularization | LB SGD, ResNet-18 · CIFAR10, CIFAR100, Tiny-ImageNet, SVHN | test accuracy | Hurt |
| Replace loss-output-gradient weighting with unit Jacobian regularization | LB SGD, ResNet-18 · CIFAR10, CIFAR100, Tiny-ImageNet, SVHN | test accuracy | Hurt |
| Increase micro-batch size from 128 to 2560 for gradient-norm regularization (ablation) | GN, ResNet-18 · CIFAR100 and Tiny-ImageNet | test accuracy: 70.22 -> 64.23; 37.78 -> 31.96 | Hurt |
| Increase micro-batch size from 128 to 2560 for Fisher trace regularization (ablation) | FT, ResNet-18 · CIFAR100 and Tiny-ImageNet | test accuracy: 71.19 -> 65.44; 40.25 -> 37.71 | Hurt |
| Use single-sample micro-batch gradient norm instead of the average micro-batch norm (ablation) | Large-batch regularized training · CIFAR10 and CIFAR100 | test accuracy | No clear effect |
| Use larger sample micro-batch size (ablation) | Sample gradient norm regularization · CIFAR10 | test accuracy | Hurt |
| Use external gradient grafting | Large-batch SGD · CIFAR10 | test accuracy: 89.99 -> 92.12 | Helped |
| Use external gradient grafting | Large-batch SGD · CIFAR100 | test accuracy: 66.17 -> 68.3 | Hurt |
| Use iterative gradient grafting | Large-batch SGD · CIFAR10 and CIFAR100 | test accuracy: 89.99 -> 92.16; 66.17 -> 68.4 | Mixed |
| Use normalized gradient descent | Large-batch SGD · CIFAR10 and CIFAR100 | test accuracy: 89.99 -> 92.10; 66.17 -> 66.83 | Mixed |
Closest papers by meaning
- Stochastic Training is Not Necessary for Generalization (ICLR 2022 · similarity 0.93)
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (ICLR 2017 · similarity 0.93)
- On the Origin of Implicit Regularization in Stochastic Gradient Descent (ICLR 2021 · similarity 0.92)
- Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions (NeurIPS 2022 · similarity 0.92)
- The Benefits of Implicit Regularization from SGD in Least Squares Problems (NeurIPS 2021 · similarity 0.92)
- SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs (NeurIPS 2021 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.