Research map / Learning theory and algorithms / Sample complexity
Fast Equilibrium of SGD in Generic Situations
Zhiyuan Li, Yi Wang, Zhiren Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies the Fast Equilibrium conjecture for SGD on scale-invariant normalized networks, proving that equilibrium is reached in roughly O~(1/(eta*lambda)) steps under generic conditions.
Problem
Normalization layers make networks scale-invariant, and training with SGD, learning rate eta and weight decay lambda, has unexpected dynamics; the Fast Equilibrium conjecture (Li et al., 2020) claims mixing to equilibrium in about O~(1/(eta*lambda)) steps.
Approach
Analyzes the mixing time of SGD with weight decay on scale-invariant normalized networks in generic situations, providing support for the Fast Equilibrium conjecture rather than relying on the specific setting studied earlier. Details beyond the abstract are not confirmed here.
Experimental setup
Builds on: Fast Equilibrium conjecture (Li et al., 2020)
Closest papers by meaning
- Fast Mixing of Stochastic Gradient Descent with Normalization and Weight Decay (NeurIPS 2022 · similarity 0.89)
- Three-stage Evolution and Fast Equilibrium for SGD with Non-degerate Critical Points (ICML 2022 · similarity 0.87)
- Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate (NeurIPS 2020 · similarity 0.86)
- The Heavy-Tail Phenomenon in SGD (ICML 2021 · similarity 0.84)
- Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks (ICLR 2018 · similarity 0.84)
- Does SGD really happen in tiny subspaces? (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
