Research map / Optimization / Implicit bias and generalization
Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that steepest descent (GD, coordinate descent, sign descent, spectral/modular-norm descent) on homogeneous neural networks with exponential loss increases an algorithm-dependent soft geometric margin after training accuracy is perfect, and that limit points align with KKT points of the corresponding norm-based margin maximization problem.
Problem
Understanding the implicit bias of steepest descent algorithms (beyond gradient descent, e.g., sign descent, coordinate descent, Shampoo-like spectral descent, and Adam-related methods) in deep homogeneous neural networks, which were previously analyzed mainly for l2 margin maximization.
Approach
Generalizes margin-maximization analysis of gradient flow to steepest flow with respect to an arbitrary norm: defines a soft margin normalized by the algorithm's norm, proves it is non-decreasing after separation, and shows limit points are KKT points of the corresponding norm-squared margin problem, using a direct limit-point argument rather than approximate-KKT sequences.
Key results
- Soft margin is non-decreasing after reaching perfect training accuracy; in teacher-student experiments each algorithm increases its corresponding geometric margin (SD attains larger l1 margin than GD and CD).
- Switching from GD to CD or SD after separation changes the final l_inf margin, but gives at most marginal test-accuracy changes, while full CD runs generalize best, suggesting early-stage dynamics matter for generalization.
- On MNIST, Adam with default epsilon initially behaves like sign descent, increasing gamma_1, before shifting toward l2-margin growth; Shampoo attains the largest spectral margin among Shampoo, Adam and GD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with each algorithm beyond perfect training accuracy | Steepest descent algorithms · Teacher-student synthetic data | algorithm-specific geometric margin | Helped |
| Decrease initialization magnitude | Coordinate descent (CD) · Teacher-student synthetic data | gamma_infinity margin | Helped |
| Use coordinate descent instead of gradient descent | One-hidden-layer ReLU network · Teacher-student synthetic data | test accuracy | Helped |
| Switch from gradient descent to coordinate descent after perfect training accuracy | Gradient descent (GD) · Teacher-student synthetic data | test accuracy | Helped |
| Switch from gradient descent to sign descent after perfect training accuracy | Gradient descent (GD) · Teacher-student synthetic data | test accuracy | Hurt |
| Switch from gradient descent to sign descent after perfect training accuracy | Gradient descent (GD) · Teacher-student synthetic data | gamma_infinity margin | Hurt |
| Use sign descent instead of gradient descent | Two-layer neural network · MNIST digit pair | gamma_1 margin | Helped |
| Use gradient descent instead of sign descent | Two-layer neural network · MNIST digit pair | gamma_2 margin | Helped |
| Use default Adam hyperparameters | Two-layer neural network · MNIST digit pair | gamma_1 margin | Mixed |
| Increase Adam precision epsilon | Adam · MNIST digit pair | gamma_1 margin | Helped |
| Use Shampoo instead of Adam or gradient descent | Two-layer neural network with frozen second layer · MNIST digit pair | spectral margin | Helped |
Surprising findings
- Adam with default hyperparameters initially increases the l1-type margin like sign descent, then decreases it to slowly increase the l2 margin, and the second phase would only be visible after loss below about 1e-7.
- Switching late in training to a different steepest descent variant yields only marginal generalization changes despite changing the final margin, questioning a causal link between geometric margin and generalization.
Limitations
- Analysis is for steepest flow (infinitesimal learning rate), not discrete steps with finite learning rate.
- Only guarantees convergence of limit points to KKT points, not finite-time or global optimality; KKT points need not be global maxima.
- Experiments are small-scale (one-hidden-layer networks, MNIST digit pairs), with no large-scale validation.
Experimental setup
Datasets: MNIST (pair of digits) synthetic Gaussian teacher-student data
Benchmarks: MNIST (digit pair) synthetic teacher-student test accuracy
Metrics: geometric margins gamma_1, gamma_2, gamma_inf, spectral marg test accuracy train accuracy
Baselines: gradient descent Adam Shampoo
Builds on: Lyu and Li (2020) gradient flow margin maximization in homogeneous networks Steepest descent (Bernstein et al. 2020 / Pethick et al. modular norm) Adam Shampoo
Closest papers by meaning
- The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks (ICML 2026 · similarity 0.92)
- Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks (ICML 2025 · similarity 0.92)
- The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks (ICML 2021 · similarity 0.90)
- The Implicit Bias of Adam on Separable Data (NeurIPS 2024 · similarity 0.90)
- On the Implicit Bias of Adam (ICML 2024 · similarity 0.89)
- The Rich and the Simple: On the Implicit Bias of Adam and SGD (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.