Research map / Optimization / Optimizers
How Memory in Optimization Algorithms Implicitly Modifies the Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a general technique that replaces an optimizer with memory (momentum variables) by a memoryless iteration plus a loss-perturbation correction term, and uses it to show Lion avoids the implicit anti-regularization that AdamW's memory induces.
Problem
Understanding the implicit bias introduced by memory (exponentially decaying history dependence) in first-order deep learning optimizers such as momentum GD, AdamW, and Lion, and how it affects generalization.
Approach
Replace all past iterates by the current one and add a memory-induced correction term (a function of the current iterate) to obtain a memoryless iteration with provable O(lr^2) global error; the correction is interpreted as a perturbation of the loss that reveals implicit regularization or anti-regularization.
Key results
- Derives memoryless iteration with global approximation error O(lr^2) on finite time horizon under exponential memory decay assumption.
- Recovers the known modified loss for heavy-ball momentum GD: loss plus lr*beta/(2(1-beta)^2) * ||grad||^2 (implicit regularization).
- Shows AdamW's memory implicitly anti-penalizes the one-norm of the gradient when weight decay is small, whereas Lion's correction term is small for small epsilon, offering a theoretical explanation for Lion's better generalization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add exponentially decaying memory to full-batch AdamW (assuming β₂ > β₁ and sufficiently small weight decay) | AdamW · Full-batch optimization | implicit regularization | Hurt |
| add memory to Lion | Lion · Full-batch optimization | generalization | Helped |
| add momentum memory | Heavy-ball momentum GD · Optimization dynamics | implicit regularization | Helped |
| add mini-batch sampling noise to momentum training | Mini-batch GD with momentum · Mini-batch training | implicit regularization | Helped |
Surprising findings
- Lion does not have the implicit anti-regularization that AdamW's memory induces, contrary to the regularization view of AdamW's memory effect.
Limitations
- Approximation bounds require sufficiently small learning rate.
- Mini-batch noise effects on Lion vs Adam are not fully characterized and are left to future work.
- Large-scale empirical testing is out of scope due to compute constraints; only a preliminary illustration is given.
Experimental setup
Datasets: CIFAR-10
Benchmarks: CIFAR-10
Metrics: training accuracy
Baselines: GD with momentum Nesterov accelerated gradient AdamW Lion Signum
Builds on: heavy-ball momentum AdamW Lion modified equations Signum
Closest papers by meaning
- On the Implicit Bias of Adam (ICML 2024 · similarity 0.87)
- Ascent Fails to Forget (NeurIPS 2025 · similarity 0.86)
- Estimating Implicit Regularization in Deep Learning (NeurIPS 2026 · similarity 0.86)
- To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.86)
- When Will Gradient Regularization Be Harmful? (ICML 2024 · similarity 0.85)
- Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.