AI Paper Atlas

Research map / Optimization / Optimizers

Why are Adaptive Methods Good for Attention Models?

Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, Suvrit Sra

NeurIPS 2020 · Optimization › Optimizers · approach: Adam convergence and adaptive learning rates · 53 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace Adam with ACClipBERT Base 12 layers · BERT pretraining validationval loss: 1.718 -> 1.615Improved
replace Adam with ACClipBERT Large 24 layers · BERT pretraining validationmasked LM accuracy: 70.56 -> 70.97Improved
replace Adam with ACClipBERT Base 6 layers · SQuAD v1.1 devF1: 84.79 -> 85.87Improved
replace Adam with ACClipBERT Base 12 layers · SQuAD v1.1 devF1: 88.61 -> 88.82Improved
replace Adam with ACClipBERT Large 24 layers · SQuAD v1.1 devexact match: 83.94 -> 84.93Improved
add global gradient clipping to SGD momentum (ablation)SGD momentum · BERT Base pretraininglossImproved
add global clipping to SGD, but still not matching Adam (ablation)clipped SGD · BERT Base pretraininglossNo clear effect
replace global clipping with coordinate-wise clipping (ablation)clipped SGD · Transformer-XL PTBtraining lossImproved
add online moment estimation for the clipping threshold (ablation)coordinate-wise clipping · Transformer-XL PTBtraining lossImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy