Research map / Optimization / Implicit bias and generalization
Gradient Starvation: A Learning Proclivity in Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ERM with Spectral Decoupling | Neural network trained on Colored MNIST · Colored MNIST test environment | test accuracy: 23.7 % (±0.8) -> 68.4 % (±1.2) | Helped |
| Replace ERM with Spectral Decoupling | Neural network trained on Colored MNIST · Colored MNIST training environments | train accuracy: 91.1 % (±0.4) -> 70.0 % (±0.9) | Hurt |
| Use Spectral Decoupling instead of vanilla cross-entropy (ERM) | CNN · CIFAR-2, CIFAR-10, and CIFAR-100 perturbed test sets (ε_PGD=0.05) | accuracy under perturbation | Helped |
| Use Spectral Decoupling instead of vanilla cross-entropy (ERM) | CNN · CIFAR-2, CIFAR-10, and CIFAR-100 original test sets | test accuracy | No clear effect |
| Use Spectral Decoupling instead of ERM | ResNet-50 · CelebA blond-versus-dark hair classification, blond-male worst group | worst-group accuracy | Helped |
| Add Spectral Decoupling | Two-Moon classifier · Two-Moon classification | input-space margin | Helped |
| Increase the strength of one feature when features are coupled | Cross-entropy-trained linearized network · Coupled two-feature setting | response to the weaker feature | Hurt |
| Replace weight decay on parameters with an L2 penalty on logits | Cross-entropy-trained network · Feature-learning dynamics | feature coupling | Helped |
| Use weight decay, dropout, or batch normalization instead of Spectral Decoupling | Two-Moon classifier · Two-Moon classification | input-space margin | No clear effect |
| Train longer, use Adam, or change the architecture or coordinate system instead of Spectral Decoupling | Two-Moon classifier · Two-Moon classification | learning a curved decision boundary | No clear effect |
| Use Spectral Decoupling while retaining color-feature learning | Neural network trained on Colored MNIST · Colored MNIST with reversed color-label correlation and digit features present | accuracy | Helped |
Closest papers by meaning
- How Memory in Optimization Algorithms Implicitly Modifies the Loss (NeurIPS 2025 · similarity 0.87)
- Grokking as the transition from lazy to rich training dynamics (ICLR 2024 · similarity 0.87)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.87)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.87)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.87)
- Feature Contamination: Neural Networks Learn Uncorrelated Features and Fail to Generalize (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.