Research map / Optimization / Implicit bias and generalization
The Implicit Bias of Adam on Separable Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that full-batch Adam with no stability constant, on linearly separable logistic/exponential-loss linear classification, converges in normalized direction to the maximum ℓ∞-margin classifier, with polynomial-time rates for diminishing learning rates.
Problem
Characterizing the implicit bias of Adam (without the stability constant epsilon) on linearly separable binary classification, where gradient descent is known to converge to the maximum ℓ2-margin solution.
Approach
Analyzes Adam's moment estimates entry-wise with bounds scaled by the gradient mass G(w)=-(1/n)Σℓ'(margin), showing the normalized margin converges to the ℓ∞ max-margin, distinct from GD's ℓ2 bias, and gives convergence rates for learning rates η_t=(t+2)^{-a}.
Key results
- Adam without epsilon converges to the maximum ℓ∞-margin solution, while GD and GDM converge to the maximum ℓ2-margin solution in simulations with n=50, d=50 and 10^6 iterations.
- For η_t=Θ(t^{-a}) with a<1, the margin gap decays polynomially, e.g. O(t^{-a/2}) for a<2/3 and O(t^{1-a}) for 2/3<a<1, versus O(log log t / log t) for GD.
- Theorem gives the margin convergence for logistic and exponential loss under diminishing learning rates with sum η_t = ∞ and η_t → 0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Adam without the stability constant ε instead of GD or GDM | Linear classification on linearly separable data · Synthetic separable data (logistic loss) | normalized ℓ∞-margin | Helped |
| Use Adam without the stability constant ε instead of GD or GDM | Linear classification on linearly separable data · Synthetic separable data (logistic loss) | normalized ℓ2-margin | Hurt |
| Add stability constant ε = 10^{-8} to Adam (ablation) | Adam (ε = 0) · Synthetic separable data (logistic loss) | normalized ℓ∞-margin | No clear effect |
| Add stability constant ε = 10^{-8} to Adam (ablation) | Adam (ε = 0) · Synthetic separable data (logistic loss) | normalized ℓ2-margin | No clear effect |
| Use a learning-rate schedule η_t = Θ(t^{-a}) with a < 1 (ablation) | Adam (ε = 0) · Synthetic separable data | normalized ℓ∞-margin gap convergence behavior | Helped |
| Use a learning-rate schedule η_t = Θ(t^{-1}) (ablation) | Adam (ε = 0) · Synthetic separable data | normalized ℓ∞-margin gap convergence behavior | Mixed |
| Use learning rate η_t = Θ(t^{-0.3}) rather than the other tested schedules (ablation) | Adam · Selected synthetic separable dataset | normalized ℓ∞-margin convergence speed | Helped |
| Use η_t = Θ(t^{-0.3}) across different datasets (ablation) | Adam · Different learning tasks | normalized ℓ∞-margin convergence speed | Mixed |
| Train with logistic loss rather than exponential loss (ablation) | Adam (ε = 0) · Linearly separable binary linear classification | normalized ℓ∞-margin convergence | No clear effect |
Surprising findings
- Adam's implicit bias is the ℓ∞ max-margin, unlike GD's ℓ2 max-margin, so the stability constant epsilon changes the regime but the simulations show epsilon=1e-8 behaves similarly to epsilon=0.
Limitations
- Analysis restricted to full-batch Adam and linear models; extension to homogeneous neural networks left open.
- Rates are worst-case bounds with large constants and t0 that is only polynomial in problem parameters without explicit formula.
- Requires assumptions on learning rates and initialization gradient magnitude (rho).
Experimental setup
Datasets: synthetic Gaussian data (n=50, d=50)
Benchmarks: synthetic linearly separable data
Metrics: normalized l_infty margin normalized l_2 margin margin gap vs iterations
Baselines: gradient descent gradient descent with momentum Adam with epsilon=1e-8
Builds on: Adam gradient descent implicit bias (Soudry et al.) AdamW implicit bias AdaGrad implicit bias
Closest papers by meaning
- The Rich and the Simple: On the Implicit Bias of Adam and SGD (NeurIPS 2025 · similarity 0.91)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.90)
- On the Implicit Bias of Adam (ICML 2024 · similarity 0.90)
- Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks (ICLR 2025 · similarity 0.90)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.88)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.