Research map / Optimization / Optimizers
Understanding Adam Requires Better Rotation Dependent Assumptions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows Adam's performance on transformers degrades under random parameter-space rotations (especially global), that SVD-based rotations improve it, that existing rotation-dependent assumptions (L_inf gradient bound, L_inf smoothness/(1,1)-norm, block-diagonal Hessian) fail to explain this, and that update orthogonality (coefficient of variation of singular values) tracks performance.
Problem
Adam is not rotation-equivariant unlike SGD, yet existing theoretical explanations of Adam's advantage over SGD rely on rotation-invariant assumptions that cannot capture its basis dependence in transformer training.
Approach
Empirically map Adam's sensitivity across rotation scopes (neuron/output/input/layer/global) and structured SVD rotations, test rotation-dependent assumptions against these results, and identify scaled orthogonality of layer updates (CV of singular values) as a quantity that aligns with performance under rotation.
Key results
- Global random rotation slows GPT-2 training by 16% and ViT by 96% relative to unrotated AdamW.
- SVD-based rotations (recomputed every 250 steps) give a sizeable improvement in GPT-2 training over unrotated Adam.
- Coefficient of variation of singular values of Adam layer updates tracks performance across rotations: SVD lowest, global highest; existing L_inf gradient bound and block-diagonal Hessian assumptions do not match.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply a global random rotation | AdamW on GPT-2 · OpenWebText | training speed: 16% slowdown | Hurt |
| Apply a global random rotation | AdamW on ViT/S · ImageNet-1K | training speed: 96% slowdown | Hurt |
| Apply full-rank SVD rotations, recomputed every 250 steps | AdamW on GPT-2 · OpenWebText | performance | Helped |
| Apply random output-wise rotations | AdamW on GPT-2 · OpenWebText | performance | Mixed |
| Apply broader-scope random rotations rather than output-wise rotations | AdamW on transformers · GPT-2 and ViT/S | performance | Hurt |
| Apply random rotations | AdamW on ResNet-50 · ImageNet-1K | performance | No clear effect |
| Set β₂ to 0.9 or 0.99 (ablation) | AdamW with global rotation on GPT-2 · OpenWebText | performance | No clear effect |
| Increase the random-rotation block dimension (ablation) | AdamW on GPT-2 · OpenWebText | training loss | No clear effect |
| Rotate only embedding layers layer-wise rather than output-wise (ablation) | AdamW on GPT-2 · OpenWebText | performance | Hurt |
| Use output-wise rather than layer-wise rotation on value layers (ablation) | AdamW on GPT-2 · OpenWebText | performance | No clear effect |
Surprising findings
- Off-diagonal Hessian blocks (other layers) contribute more to gradient variation than within-neuron blocks, challenging the block-diagonal Hessian assumption.
- Output-wise rotations cause no degradation and slightly improve GPT-2, while ResNet-50 shows minimal sensitivity to rotations.
Limitations
- No theoretical guarantees linking update orthogonality to convergence; the quantity is empirical.
- SVD rotation is a diagnostic rather than a practical optimizer proposal.
- Results on ResNets are less conclusive given the smaller SGD-Adam gap; limited seeds due to compute.
Experimental setup
Datasets: OpenWebText ImageNet-1K
Benchmarks: ImageNet-1K OpenWebText
Metrics: training loss validation loss top-1 validation accuracy Hessian (1,1)-norm L_inf gradient bound coefficient of variation of singular values
Baselines: AdamW SGD with momentum Muon SOAP
Builds on: Adam AdamW GaLore Shampoo SOAP Muon
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.88)
- Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity (ICLR 2025 · similarity 0.88)
- In Search of Adam’s Secret Sauce (NeurIPS 2025 · similarity 0.88)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.87)
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · similarity 0.87)
- The Implicit Bias of Adam on Separable Data (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.