AI Paper Atlas

Research map / Optimization / Nonconvex and smooth optimization

Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Rate

Zhiqi Bu, Shiyun Xu, Jialin Mao

ICLR 2026 · Optimization › Nonconvex and smooth optimization · approach: Coordinate and Adam convergence analysis · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
reduce batch size from 512 to 64 (ablation)O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation)estimated L_inf: 2.8298 -> 2.9670Hurt
reduce batch size from 512 to 64 (ablation)O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation)estimated Q: 23.9094 -> 39.5403Hurt
reduce batch size from 512 to 64 (ablation)O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation)R^2 of linear fit: 0.998 -> 0.996No clear effect
change random seed from 1337 to 8888 (ablation)O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation)R^2 of linear fit: 0.998 -> 0.995No clear effect
use gradient clipping 1.0 instead of 0.0 (ablation)O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation)estimated L_inf: 2.8298 -> 2.8207No clear effect
lower momentum from 0.95 to 0.9 (ablation)O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation)estimated L_inf: 2.8298 -> 2.8432No clear effect
remove weight decay (0.01 to 0.0) (ablation)O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation)estimated Q: 23.9094 -> 25.1154No clear effect
remove weight decay (0.01 to 0.0) (ablation)O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation)R^2 of linear fit: 0.998 -> 0.999No clear effect
increase model size from 0.074B to 12.56BChinchilla runs · Chinchilla runs, AdamW cosinefitted L_inf: 2.825 -> 2.053Improved
train with overfitting (after 50 epochs) on CIFAR10 ResNet50O(1/sqrt(T)) convergence law · ResNet50 CIFAR10loss convergenceHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy