Research map / Optimization / Optimizers
Noise Is Not the Main Factor Behind the Gap Between Sgd and Adam on Transformers, But Sign Descent Might Be
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase batch size toward full batch | Adam · Transformer language tasks including PTB, WikiText-2, and SQuAD | training loss per iteration | Helped |
| increase batch size beyond a threshold | plain SGD · Transformer language tasks | training loss per iteration | No clear effect |
| increase batch size toward full batch | SGD versus Adam · Transformer language tasks | training loss | Mixed |
| replace the gradient update with its element-wise sign | gradient descent · Transformer language tasks, full batch | training loss | Helped |
| replace the gradient update with its element-wise sign | gradient descent · Transformer language tasks, small batch | training loss | Hurt |
| normalize the gradient by its L2 norm | gradient descent · Transformer language tasks | training loss | Helped |
| use element-wise sign updates instead of L2-normalized gradient updates | normalized GD · Transformer language tasks, full batch | training loss | Helped |
| add heavy-ball momentum | SGD, Adam, sign descent, and normalized GD · training across batch sizes | convergence speed | Helped |
| remove stochastic gradient noise by using full-batch training | SGD versus Adam · Transformer language tasks | training loss | Mixed |
| disable dropout (ablation) | full-batch Transformer training · PTB and WikiText-2 | training loss | Helped |
| add heavy-ball momentum (ablation) | sign descent · Transformer language tasks, full batch | training loss | Helped |
Closest papers by meaning
- On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent (ICLR 2025 · similarity 0.90)
- In Search of Adam’s Secret Sauce (NeurIPS 2025 · similarity 0.90)
- Why Transformers Need Adam: A Hessian Perspective (NeurIPS 2024 · similarity 0.90)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.90)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.90)
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.