Research map / Architectures and efficiency / RNNs and normalization
GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training
Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W Ronny Huang, Tom Goldstein
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GradInit initialization instead of Kaiming | VGG-19 w/ BN · CIFAR-10 | best test accuracy (%): 94.4 -> 95.1 | Improved |
| use GradInit instead of Kaiming for 1202-layer ResNet | ResNet-1202 w/ BN · CIFAR-10 | best test accuracy (%): 94.4 -> 96.2 | Improved |
| use GradInit instead of Kaiming for ResNet-50 on ImageNet | ResNet-50 w/ BN · ImageNet | top-1 accuracy (%): 75.9 -> 76.2 | Improved |
| use GradInit without warmup for Post-LN Transformer trained with SGD vs Admin | Post-LN Transformer · IWSLT-14 De-En | BLEU: 33.7 -> 35.6 | Improved |
| also rescale BN scale parameters (vs fixing BN) (ablation) | GradInit · CIFAR-10 | best test accuracy (%): 94.6 -> 95.1 (VGG-19 BN); 94.7 -> 95.4 (ResNet-110 BN) | Improved |
| also rescale linear layers (vs only BN) (ablation) | GradInit · CIFAR-10 | best test accuracy (%): 94.4 -> 95.1 (VGG-19 BN) | Improved |
| use GradInit's one-step-loss objective instead of simply learning scalars (ablation) | GradInit · CIFAR-10 | best test accuracy (%): 10.0 -> 95.1 (VGG-19 BN); 94.7 -> 95.4 (ResNet-110 BN) | Improved |
| use gradient-norm constraint (gamma=1) instead of gradient penalty lambda=1 (ablation) | GradInit · CIFAR-10 | best test accuracy (%): 74.7 -> 94.7 (VGG-19 w/o BN) | Improved |
| use 50% overlap between S and S~ instead of independent batches (ablation) | GradInit · CIFAR-10 | first-epoch accuracy (%): 21.9 -> 29.3 | Improved |
| use 50% overlap instead of identical batches (S~ = S) (ablation) | GradInit · CIFAR-10 | best test accuracy (%): 94.5 -> 94.7 | Improved |
| keep gradient norm constraint when S~ = S (ablation) | GradInit · CIFAR-10 | best test accuracy (%): 93.70 -> 94.78 | Improved |
| add skip scaling w_skip to GradInit Transformer (ablation) | GradInit with Adam · IWSLT-14 De-En | BLEU: 36.1 -> 36.0 | Mixed |
Later papers that compared with this method
1 later accepted paper reports Gradinit as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Towards Theoretically Inspired Neural Initialization Optimization (NeurIPS 2022 · better on 15 of 15 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- IDInit: A Universal and Stable Initialization Method for Neural Network Training (ICLR 2025 · similarity 0.85)
- Structured Initialization for Vision Transformers (NeurIPS 2025 · similarity 0.84)
- Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights (NeurIPS 2024 · similarity 0.84)
- A Multi-Level Framework for Accelerating Training Transformer Models (ICLR 2024 · similarity 0.84)
- MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent (ICML 2024 · similarity 0.84)
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
