Research map / Time series / Foundation models for time series
GradientStabilizer: Fix the Norm, Not the Gradient
Tianjin Huang, Zhangyang Wang, Haotian Hu, Zhenyu Zhang, Gaojie Jin, Xiang Li, Li Shen, Jiaxing Shang, Tianlong Chen, Ke Li, Lu Liu, Qingsong Wen and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GradientStabilizer to Adam | Adam · C4 FP4 training, LLaMA-350M | validation perplexity: 21.57 -> 18.89 | Improved |
| add GradientStabilizer to AdamW | AdamW · C4 FP4 training, LLaMA-130M | validation perplexity: 28.72 -> 26.66 | Improved |
| add GradientStabilizer to AdamW | AdamW · C4 FP16 training, LLaMA-130M | validation perplexity: 24.31 -> 23.14 | Improved |
| add Norm Clip to Adam | Adam · C4 FP4 training, LLaMA-350M | validation perplexity: 21.57 -> 22.29 | Hurt |
| add AGC to AdamW | AdamW · ImageNet-1K, ViT-B | top-1 accuracy: 79.3 -> 38.5 | Hurt |
| add GradientStabilizer to AdamW | AdamW · ImageNet-1K, ViT-B | top-1 accuracy: 79.3 -> 79.6 | Improved |
| add GradientStabilizer to Adam | Adam · ImageNet-1K, ViT-B, weight decay 1e-4 | top-1 accuracy: 59.3 -> 78.7 | Improved |
| add gradient clipping (Norm Clip) to Adam | Adam · ImageNet-1K, ViT-B, weight decay 5e-4 | top-1 accuracy: 52.9 -> 20.6 | Hurt |
| add GradientStabilizer to Adam at weight decay 1e-4 | Adam · ImageNet-1K, ConvNeXt-T, weight decay 1e-4 | top-1 accuracy: 35.2 -> 78.8 | Improved |
| add GradientStabilizer to Adam-Mini | Adam-Mini · C4 FP4 training, LLaMA-130M | validation perplexity: 31.66 -> 27.75 | Improved |
| add GradientStabilizer to Lion | Lion · C4 FP4 training, LLaMA-350M | validation perplexity: 23.59 -> 22.75 | Improved |
| vary decay gamma1 in [0.5, 0.8] and gamma2 in [0.95, 0.99999] (ablation) | GradientStabilizer · LLaMA-130M FP4 C4 | final validation loss: variation less than 0.02 | No clear effect |
Closest papers by meaning
- Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size (NeurIPS 2025 · similarity 0.87)
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.86)
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.86)
- Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.86)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
