Research map / Optimization / Implicit bias and generalization
Characterizing Implicit Bias in Terms of Optimization Geometry
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mirror descent (dual-space updates) with a strongly convex potential for losses with unique finite root, initialized at argmin psi | Mirror descent on underdetermined linear regression · Underdetermined linear models, unique-finite-root losses | limit point w_inf (minimum Bregman divergence global minimizer) | Helped |
| Add primal momentum to mirror descent instead of dual momentum (ablation) | Mirror descent with entropy potential · Example 2 toy dataset, unique-finite-root squared loss | limit point depends on momentum parameters | Hurt |
| Use finite step size in natural gradient descent instead of infinitesimal step size (ablation) | Natural gradient descent with entropy potential metric · Example 3 toy dataset, squared loss | limit point w_inf | Hurt |
| Steepest descent with respect to a generic norm (l4/3) with finite step size, for squared loss (ablation) | Steepest descent · Example 4 toy dataset, squared loss | limit point depends on step size and differs from min-norm solution even as eta- | Hurt |
| Steepest descent with respect to generic norm on exponential loss (separable data), bounded step sizes | Steepest descent · Separable linear classification, exponential loss | limit direction of iterates | Helped |
| Use AdaGrad (diagonal adaptive preconditioning) on separable exponential loss problems (ablation) | AdaGrad · Separable linear classification, exponential loss | limit direction depends on initial conditions and step size | Hurt |
| Factorized parameterization W=UU^T with gradient descent on U under exponential loss (vs. gradient descent on W directly) | Gradient descent on matrix factorization · Separable matrix classification, exponential loss | limit direction is max-margin with unit nuclear norm, independent of initializat | Helped |
| Factorized parameterization with squared loss, finite step size and initialization | Gradient descent on matrix factorization · Underdetermined squared loss | implicit bias toward minimum nuclear norm | Mixed |
Closest papers by meaning
- Implicit Bias of the Step Size in Linear Diagonal Neural Networks (ICML 2022 · similarity 0.90)
- The Implicit Bias of AdaGrad on Separable Data (NeurIPS 2019 · similarity 0.89)
- Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks (ICLR 2025 · similarity 0.88)
- Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data (NeurIPS 2025 · similarity 0.87)
- Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regression (ICLR 2025 · similarity 0.87)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.