Research map / Architectures and efficiency / RNNs and normalization
GradMax: Growing Neural Networks using Gradient Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GradMax growth instead of random growth | WRN-28-1 · CIFAR-10 | test accuracy: 90.6$\pm$0.2 -> 91.1$\pm$0.1 | Helped |
| use GradMax growth instead of Firefly | VGG11 · CIFAR-10 | test accuracy: 84.0$\pm$0.2 -> 84.4$\pm$0.4 | Helped |
| use GradMax growth instead of random growth | WRN-28-1 · CIFAR-100 | test accuracy: 66.7$\pm$0.4 -> 66.8$\pm$0.2 | No clear effect |
| use GradMax growth instead of random growth | MobileNet-v1 · ImageNet | test accuracy: 66.9$\pm$0.3 -> 68.6$\pm$0.2 | Helped |
| use SVD initialization instead of direct gradient optimization (ablation) | GradMax · teacher-student task | training loss | Helped |
| set outgoing weights to zero and use GradMaxOpt instead of the normal formulation (ablation) | WRN-28-1 · CIFAR-10 | test accuracy: 91.1$\pm$0.1 -> 92.4$\pm$0.1 | Helped |
| add batch normalization to GradMax growth (ablation) | WRN-28-1 · CIFAR-10 | test accuracy: 91.1$\pm$0.1 -> 93.0$\pm$0.1 | Helped |
| add batch normalization while setting outgoing weights to zero (ablation) | GradMaxOpt · CIFAR-10 | test accuracy: 92.4$\pm$0.1 -> 92.9$\pm$0.2 | Helped |
| use nonzero outgoing initialization scale instead of preserving the network output exactly (ablation) | GradMax growing · ImageNet | test accuracy | Hurt |
| use fixed initialization scale instead of mean-norm normalization (ablation) | GradMax growing · ImageNet | test accuracy | No clear effect |
| grow neurons early rather than later (ablation) | GradMax growing · ImageNet | test accuracy | Helped |
| start with a larger seed network (ablation) | GradMax growing · teacher-student task | training loss | Helped |
Closest papers by meaning
- Learned Optimizers that Scale and Generalize (ICML 2017 · similarity 0.89)
- Towards Theoretically Inspired Neural Initialization Optimization (NeurIPS 2022 · similarity 0.88)
- GradMetaNet: An Equivariant Architecture for Learning on Gradients (NeurIPS 2025 · similarity 0.88)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.86)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.86)
- Optimal Rates for Generalization of Gradient Descent for Deep ReLU Classification (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.