Research map / Optimization / Implicit bias and generalization
Is SGD a Bayesian sampler? Well, almost
Chris Mingard, Guillermo Valle-Pérez, Joar Skalse, Ard A. Louis
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase batch size from 32 to 128 (ablation) | Adam, FCN on MNIST, CE loss · MNIST | mean generalisation error (%): 1.13% -> 2.20% | Hurt |
| increase batch size from 32 to 512 (ablation) | Adam, FCN on MNIST, CE loss · MNIST | mean generalisation error (%): 1.13% -> 2.67% | Hurt |
| add 4x faster learning rate at batch size 512 (ablation) | Adam, FCN on MNIST, CE loss · MNIST | mean generalisation error (%): 2.67% -> 2.14% | Improved |
| randomise 20% of training labels (ablation) | FCN on MNIST, MSE loss · MNIST | correlation between P_B and P_SGD: 13.4% and 5.80% | Hurt |
| increase training set size from 1000 to 20000 (ablation) | FCN on MNIST · MNIST | bias towards low error functions and P_B/P_SGD correlation: |S|=1000 to |S|=20000 | Improved |
| add max-pooling to the CNN (ablation) | CNN on Fashion-MNIST with Adam · Fashion-MNIST | generalisation performance | Improved |
| use NTK instead of GP posterior | FCN on MNIST · MNIST | function probabilities | Mixed |
| increase batch size under MSE loss (ablation) | FCN on MNIST, MSE loss · MNIST | generalisation error | Improved |
Closest papers by meaning
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.88)
- How Good is the Bayes Posterior in Deep Neural Networks Really? (ICML 2020 · similarity 0.88)
- Deep learning generalizes because the parameter-function map is biased towards simple functions (ICLR 2019 · similarity 0.87)
- Improving Bayesian Neural Networks by Adversarial Sampling (AAAI 2022 · similarity 0.87)
- Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning (ICML 2024 · similarity 0.86)
- Connecting the Dots: Is Mode-Connectedness the Key to Feasible Sample-Based Inference in Bayesian Neural Networks? (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
