Research map / Optimization / Implicit bias and generalization
On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity
Vincent Szolnoky, Viktor Andersson, Balazs Kulcsar, Rebecka Jörnsten
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MGS trace penalty | MGS · MNIST LeNet-5 80% noise | test accuracy: 23.9 -> 74.5 | Improved |
| add MGS penalty vs dropout | MGS · MNIST LeNet-5 80% noise | test accuracy: 39.5 -> 74.5 | Improved |
| add MGS penalty | MGS · MNIST LeNet-5 0% noise | test accuracy: 89.6 -> 95.2 | Improved |
| add MGS penalty | MGS · MNIST FCN 80% noise | test accuracy: 24.8 -> 51.4 | Improved |
| add dropout | baseline · MNIST FCN 0% noise | test accuracy: 82.1 -> 79.5 | Hurt |
| add MGS penalty | MGS · Facial Keypoints LeNet-5 noise 0 | test loss: 68.2 -> 14.1 | Improved |
| add MGS penalty vs dropout | MGS · Facial Keypoints LeNet-5 noise 30 | test loss: 19.7 -> 17.8 | Improved |
| add MGS penalty | MGS · Facial Keypoints FCN noise 0 | test loss: 281.1 -> 10.6 | Improved |
| add dropout | baseline · Facial Keypoints FCN noise 0 | test loss: 281.1 -> 208.2 | Mixed |
| change learning rate in training-parameter robustness test | MGS · MGS on corrupted MNIST | test accuracy | Hurt |
Closest papers by meaning
- Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee (ICLR 2020 · similarity 0.87)
- Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel (NeurIPS 2019 · similarity 0.87)
- Towards Understanding Regularization in Batch Normalization (ICLR 2019 · similarity 0.86)
- Network-to-Network Regularization: Enforcing Occam's Razor to Improve Generalization (NeurIPS 2021 · similarity 0.86)
- Learning Neural Networks with Adaptive Regularization (NeurIPS 2019 · similarity 0.86)
- A Unified Approach to Controlling Implicit Regularization via Mirror Descent (JMLR 2023 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
