Research map / Optimization / Implicit bias and generalization
Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime
Beomhan Baek, Minhak Song, Chulhee Yun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use batch size 1 (incremental) instead of full batch for Adam (ablation) | Adam · Gaussian data (N=10, d=50) | direction of convergence (cosine similarity to max-margin solutions) | Mixed |
| use incremental Adam on scaled Rademacher data instead of full-batch Adam (ablation) | Adam · Scaled Rademacher data (4 points, d=4) | direction of convergence | Mixed |
| increase batch size toward full batch for incremental Adam (ablation) | Adam · Gaussian data (N=10, d=50) | cosine similarity to l_inf-max-margin | Improved |
| vary beta1 in {0.1, 0.5, 0.9} with beta2=0.95 (ablation) | incremental Adam, batch size 1 · Gaussian data (N=10, d=50) | limit direction | No clear effect |
| lower beta2 to 0.1 or 0.5 instead of 0.95 (ablation) | incremental Adam, batch size 1 · Gaussian data (N=10, d=50) | agreement with fixed-point solution | Hurt |
| use Signum with beta=0.99 and smaller batch sizes (5, 2, 1) (ablation) | Signum · Gaussian data (N=10, d=50) | convergence to l_inf-max-margin | No clear effect |
Closest papers by meaning
- The Implicit Bias of Adam on Separable Data (NeurIPS 2024 · similarity 0.94)
- The Rich and the Simple: On the Implicit Bias of Adam and SGD (NeurIPS 2025 · similarity 0.90)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.89)
- Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks (ICLR 2025 · similarity 0.89)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.88)
- Implicit Bias of AdamW: $\ell_\infty$-Norm Constrained Optimization (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
