Research map / Optimization / Optimizers
GradPower: Powering Gradients for Faster Language Model Pre-Training
Jinbo Wang, Mingze Wang, Jiaqi Zhang, Wei Wang, Peng Pei, Xunliang Cai, Weinan E, Lei Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply GradPower (sign-power p=1.2) to AdamW | AdamW · LLaMA 66M to 2B on C4 and OpenWebText, cos and wsd | terminal validation loss | Improved |
| apply GradPower (p=1.2) to AdamW on MoE models | AdamW · Qwen2MoE 0.5B/1B/2B on C4, wsd | terminal validation loss: absolute gain 0.028 on 2B vs 0.022 for LLaMA-2B | Improved |
| apply GradPower to AdamW | AdamW · LLaMA-2B C4 zero-shot downstream | average accuracy: 47.72 -> 48.26 | Improved |
| add GradPower to AdamW with Blockwise LR | AdamW with Blockwise LR · LLaMA-0.2B C4 cos | terminal loss improvement: gains 0.015 + 0.03 -> 0.045 | Improved |
| apply GradPower (p=1.2) to Muon (MuonPower) | Muon · LLaMA 0.2B and 0.4B on C4, cos | validation loss | Improved |
| swap order of gradient clipping and GradPower (AdamWPower-II) (ablation) | AdamWPower · LLaMA-0.2B C4 cos | validation loss | No clear effect |
| increase batch size (lower gradient noise) when tuning power p (ablation) | AdamWPower · LLaMA-0.2B C4, batch 512 to 8192 | optimal p: optimal p falls from above 1 to possibly below 1 | Mixed |
| use p > 1 (1.1, 1.2) instead of p=1.0 at large batch (ablation) | AdamW with GradPower · CIFAR-10 ResNet-34, batch 128 | accuracy: 93.98 -> 93.15 (p=1.2) | Hurt |
| use p=0.85 instead of p=1.0 at large batch (ablation) | AdamW with GradPower · CIFAR-10 ResNet-34, batch 128 | accuracy: 93.98 -> 94.27 | Improved |
| use p=1.0 instead of p<1 at small batch (ablation) | AdamW with GradPower · CIFAR-10 ResNet-34, batch 32 | accuracy: 94.04 -> 94.30 | Improved |
| tune beta1 and beta2 for both AdamW and AdamWPower (ablation) | AdamWPower · LLaMA-0.25B OpenWebText | final validation loss: 2.832831 -> 2.828666 | Improved |
| apply sign-power to gradients | AdamWPower · LLaMA-0.25B OpenWebText | wall-clock time per step: 0.7534s -> 0.7565s (0.4% overhead) | No clear effect |
Closest papers by meaning
- Gradient Weight-normalized Low-rank Projection for Efficient LLM Training (AAAI 2025 · similarity 0.90)
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · similarity 0.90)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.89)
- GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.89)
- Progressive Residual Warmup for Language Model Pretraining (NeurIPS 2026 · similarity 0.89)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
