Research map / Architectures and efficiency / RNNs and normalization
AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping
Guoxia Wang, Shuai Li, Congliang Chen, Jinle Zeng, Jiabin Yang, Dianhai Yu, Yanjun Ma, Li Shen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AdaGC instead of GlobalGC | Llama-2 7B · spike score | spike score (%): 0.0333 -> 0.0000 | Improved |
| use AdaGC instead of GlobalGC | Qwen3-1.7B (QK-Norm) · spike score | spike score (%): 0.2842 -> 0.0053 | Improved |
| use AdaGC instead of GlobalGC | Mixtral 8x1B · zero-shot average | accuracy (%): 47.74 -> 49.01 | Improved |
| use AdaGC instead of GlobalGC | Llama-2 7B · zero-shot average | accuracy (%): 49.69 -> 51.01 | Improved |
| use AdaGC instead of GlobalGC | Llama-2 1.3B · zero-shot average | accuracy (%): 46.32 -> 46.33 | No clear effect |
| use AdaGC with eps 1e-15 instead of GlobalGC with eps 1e-15 | ERNIE 10B-A1.4B, 350B tokens · general-ability average | average score: 25.71 -> 28.19 | Improved |
| use AdaGC instead of GlobalGC (eps 1e-8) | ERNIE 10B-A1.4B, 1T tokens · general-ability average | average score: 34.52 -> 34.73 | No clear effect |
| use AdaGC with Muon optimizer instead of GlobalGC | Llama-2 1.3B · zero-shot average | accuracy (%): 47.04 -> 47.18 | No clear effect |
| use AdaGC with Lion optimizer instead of GlobalGC | CLIP ViT-Base · zero-shot average | accuracy (%): 40.65 -> 40.81 | No clear effect |
| use AdaGC instead of GlobalGC for communication | Llama-2 7B · training cost | GPU hours: 1468.2 -> 1402.4 | Improved |
| add GlobalGC warm-up with min-norm EMA initialization (ablation) | AdaGC · Llama-2 1.3B final loss | final training loss: 2.9708 vs 2.9725 for next-best | Improved |
| make clipping tensor-wise instead of a single adaptive global threshold (ablation) | AdaGC · Llama-2 1.3B | final loss: 2.9639 vs 2.9712 (0.25% higher for Global AdaGC); 1 spike vs 0 | Improved |
Closest papers by meaning
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.89)
- GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.89)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.88)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.88)
- SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training (ICLR 2025 · similarity 0.87)
- MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
