AI Paper Atlas

Research map / Optimization / Implicit bias and generalization

Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes

Maxim Kodryan, Ekaterina Lobacheva, Maksim Nakhodnov, Dmitry P. Vetrov

NeurIPS 2022 · Optimization › Implicit bias and generalization · approach: Edge-of-stability dynamics · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Train with increasing fixed effective learning rates on the sphereScale-invariant neural networks trained with projected SGD · CIFAR-10training lossMixed
Increase the ELR while remaining in the convergence regimeScale-invariant neural networks trained on the sphere · CIFAR-10test errorImproved
Increase the ELR while remaining in the convergence regimeScale-invariant neural networks trained on the sphere · CIFAR-10sharpnessImproved
Fine-tune with a decreased ELRA converged scale-invariant neural network · CIFAR-10training trajectoryNo clear effect
Fine-tune with an increased ELRA converged scale-invariant neural network · CIFAR-10sharpnessImproved
Increase ELR in the chaotic-equilibrium regimeScale-invariant neural networks trained on the sphere · CIFAR-10training lossHurt
Use low ELRs from the second regime before training with first-regime ELRsScale-invariant neural networks trained on the sphere · CIFAR-10sharpnessImproved
Use a medium starting learning rate and cosine decayResNet-18 with conventional training · CIFAR-10test accuracyImproved
Use a high starting learning rate and cosine decayResNet-18 with conventional training · CIFAR-10training lossHurt
Add data augmentationScale-invariant ConvNet · CIFAR-10training lossHurt
Increase network widthScale-invariant ConvNet · CIFAR-10training lossMixed
Add label noiseScale-invariant ConvNet · CIFAR-10sharpnessHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy