Research map / Optimization / Optimizers
Why are Adaptive Methods Good for Attention Models?
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, Suvrit Sra
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Adam with ACClip | BERT Base 12 layers · BERT pretraining validation | val loss: 1.718 -> 1.615 | Improved |
| replace Adam with ACClip | BERT Large 24 layers · BERT pretraining validation | masked LM accuracy: 70.56 -> 70.97 | Improved |
| replace Adam with ACClip | BERT Base 6 layers · SQuAD v1.1 dev | F1: 84.79 -> 85.87 | Improved |
| replace Adam with ACClip | BERT Base 12 layers · SQuAD v1.1 dev | F1: 88.61 -> 88.82 | Improved |
| replace Adam with ACClip | BERT Large 24 layers · SQuAD v1.1 dev | exact match: 83.94 -> 84.93 | Improved |
| add global gradient clipping to SGD momentum (ablation) | SGD momentum · BERT Base pretraining | loss | Improved |
| add global clipping to SGD, but still not matching Adam (ablation) | clipped SGD · BERT Base pretraining | loss | No clear effect |
| replace global clipping with coordinate-wise clipping (ablation) | clipped SGD · Transformer-XL PTB | training loss | Improved |
| add online moment estimation for the clipping threshold (ablation) | coordinate-wise clipping · Transformer-XL PTB | training loss | Improved |
Closest papers by meaning
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.90)
- To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions (ICLR 2025 · similarity 0.88)
- Efficient Distributed Optimization under Heavy-Tailed Noise (ICML 2025 · similarity 0.88)
- Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients? (NeurIPS 2026 · similarity 0.87)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.87)
- Emergence of heavy tails in homogenized stochastic gradient descent (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
