Research map / Optimization / Optimizers
Robustness to Unbounded Smoothness of Generalized SignSGD
Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, Zhenxun Zhuang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use generalized SignSGD instead of Adam | 20-layer ResNet · CIFAR-10 | test accuracy: 0.8823 ± 0.0041 -> 0.8835 ± 0.0032 | Improved |
| Use generalized SignSGD instead of Adam | 20-layer ResNet · CIFAR-10 | training loss: 0.1161 ± 0.0111 -> 0.1086 ± 0.0129 | Improved |
| Use generalized SignSGD instead of Adam | 3-layer AWD-LSTM · Penn Treebank | test perplexity: 58.9005 ± 0.3058 -> 58.9661 ± 1.5218 | No clear effect |
| Use generalized SignSGD instead of SGDClipGrad | 3-layer AWD-LSTM · Penn Treebank | training loss: 3.5774 ± 0.0081 -> 3.7928 ± 0.0425 | Hurt |
| Remove gradient clipping from Adam (ablation) | 16-layer GPT-2 · Wikitext-103 | training loss: 2.79 vs 2.76 | No clear effect |
| Remove gradient clipping from Adam (ablation) | 16-layer GPT-2 · Wikitext-103 | perplexity: 27.92 vs 27.97 | No clear effect |
| Set beta2 to 0, recovering SignSGD with Momentum (ablation) | Generalized SignSGD · Non-convex optimization under coordinate-wise (L0, L1)-smoothness | convergence bound dependence on gradient norms | Improved |
| Add momentum to SignSGD-type updates | Generalized SignSGD · Non-convex optimization under unbounded smoothness and stochastic gradients | convergence guarantee | Improved |
| Use clipping with Adam rather than Adam without clipping (ablation) | 16-layer GPT-2 · Wikitext-103 | training loss: 2.76 vs 2.79 | No clear effect |
Closest papers by meaning
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.88)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.88)
- Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients (NeurIPS 2026 · similarity 0.87)
- On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.87)
- Exact risk curves of signSGD in High-Dimensions: quantifying preconditioning and noise-compression effects (ICML 2025 · similarity 0.87)
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
