AI Paper Atlas

Research map / Optimization / Optimizers

GradPower: Powering Gradients for Faster Language Model Pre-Training

Jinbo Wang, Mingze Wang, Jiaqi Zhang, Wei Wang, Peng Pei, Xunliang Cai, Weinan E, Lei Wu

ICML 2026 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
apply GradPower (sign-power p=1.2) to AdamWAdamW · LLaMA 66M to 2B on C4 and OpenWebText, cos and wsdterminal validation lossImproved
apply GradPower (p=1.2) to AdamW on MoE modelsAdamW · Qwen2MoE 0.5B/1B/2B on C4, wsdterminal validation loss: absolute gain 0.028 on 2B vs 0.022 for LLaMA-2BImproved
apply GradPower to AdamWAdamW · LLaMA-2B C4 zero-shot downstreamaverage accuracy: 47.72 -> 48.26Improved
add GradPower to AdamW with Blockwise LRAdamW with Blockwise LR · LLaMA-0.2B C4 costerminal loss improvement: gains 0.015 + 0.03 -> 0.045Improved
apply GradPower (p=1.2) to Muon (MuonPower)Muon · LLaMA 0.2B and 0.4B on C4, cosvalidation lossImproved
swap order of gradient clipping and GradPower (AdamWPower-II) (ablation)AdamWPower · LLaMA-0.2B C4 cosvalidation lossNo clear effect
increase batch size (lower gradient noise) when tuning power p (ablation)AdamWPower · LLaMA-0.2B C4, batch 512 to 8192optimal p: optimal p falls from above 1 to possibly below 1Mixed
use p > 1 (1.1, 1.2) instead of p=1.0 at large batch (ablation)AdamW with GradPower · CIFAR-10 ResNet-34, batch 128accuracy: 93.98 -> 93.15 (p=1.2)Hurt
use p=0.85 instead of p=1.0 at large batch (ablation)AdamW with GradPower · CIFAR-10 ResNet-34, batch 128accuracy: 93.98 -> 94.27Improved
use p=1.0 instead of p<1 at small batch (ablation)AdamW with GradPower · CIFAR-10 ResNet-34, batch 32accuracy: 94.04 -> 94.30Improved
tune beta1 and beta2 for both AdamW and AdamWPower (ablation)AdamWPower · LLaMA-0.25B OpenWebTextfinal validation loss: 2.832831 -> 2.828666Improved
apply sign-power to gradientsAdamWPower · LLaMA-0.25B OpenWebTextwall-clock time per step: 0.7534s -> 0.7565s (0.4% overhead)No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy