Research map / Optimization / Nonconvex and smooth optimization
Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Rate
Zhiqi Bu, Shiyun Xu, Jialin Mao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reduce batch size from 512 to 64 (ablation) | O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation) | estimated L_inf: 2.8298 -> 2.9670 | Hurt |
| reduce batch size from 512 to 64 (ablation) | O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation) | estimated Q: 23.9094 -> 39.5403 | Hurt |
| reduce batch size from 512 to 64 (ablation) | O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation) | R^2 of linear fit: 0.998 -> 0.996 | No clear effect |
| change random seed from 1337 to 8888 (ablation) | O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation) | R^2 of linear fit: 0.998 -> 0.995 | No clear effect |
| use gradient clipping 1.0 instead of 0.0 (ablation) | O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation) | estimated L_inf: 2.8298 -> 2.8207 | No clear effect |
| lower momentum from 0.95 to 0.9 (ablation) | O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation) | estimated L_inf: 2.8298 -> 2.8432 | No clear effect |
| remove weight decay (0.01 to 0.0) (ablation) | O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation) | estimated Q: 23.9094 -> 25.1154 | No clear effect |
| remove weight decay (0.01 to 0.0) (ablation) | O(1/sqrt(T)) loss law with 1/sqrt(T)-scaled peak LR · GPT2 0.1B, Muon-NSGD, scaled peak LR (hyperparameter ablation) | R^2 of linear fit: 0.998 -> 0.999 | No clear effect |
| increase model size from 0.074B to 12.56B | Chinchilla runs · Chinchilla runs, AdamW cosine | fitted L_inf: 2.825 -> 2.053 | Improved |
| train with overfitting (after 50 epochs) on CIFAR10 ResNet50 | O(1/sqrt(T)) convergence law · ResNet50 CIFAR10 | loss convergence | Hurt |
Closest papers by meaning
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.88)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.88)
- Loss Landscape Characterization of Neural Networks without Over-Parametrization (NeurIPS 2024 · similarity 0.87)
- Stepping on the Edge: Curvature Aware Learning Rate Tuners (NeurIPS 2024 · similarity 0.87)
- Asymmetric Scaling Laws from Sparse Features (NeurIPS 2026 · similarity 0.86)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
