Research map / Architectures and efficiency / RNNs and normalization
When to Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FRAGrow, a fitting-risk-aware growth timing policy for layer-wise neural growth that uses an overfitting risk level (train minus validation accuracy) to speed up or slow down layer insertion, addressing under- and overfitting.
Problem
Neural growth (growing a small network into a large one to speed training) has a regularization effect controlled by growth timing, which can cause accuracy drops when the target model underfits, yet existing when-to-grow policies ignore this effect.
Approach
Adaptive growth interval I = I_max / (1 + exp(alpha - ORL)), where ORL is the train-minus-validation accuracy gap; underfitting models (small ORL) grow faster to reduce regularization, while overfitting models (large ORL) grow slower to strengthen regularization.
Key results
- On ImageNet (underfitting), FRAGrow beats the strongest baseline (periodic growth) by 0.47% for ResNet and 1.31% for VGG in test accuracy.
- On CIFAR10/100 (overfitting), FRAGrow matches or slightly beats the large-model baseline (e.g., ResNet CIFAR100 test error 29.14 vs 29.56) while saving about 16-20% training time.
- FRAGrow trains slower than periodic growth, saving less time (e.g., ImageNet ResNet time 100% vs 93.39% for periodic).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FRAGrow instead of periodic growth | ResNet-8-8-8-8 · ImageNet | test error (%): 24.79 -> 24.32 | Helped |
| use FRAGrow instead of periodic growth | VGG-2-2-4-4-4 · ImageNet | test error (%): 25.70 -> 24.39 | Helped |
| use FRAGrow instead of vanilla large-model training | ResNet-8-8-8-8 · CIFAR10 | test error (%): 6.66 -> 6.32 | Helped |
| use FRAGrow instead of vanilla large-model training | ResNet-8-8-8-8 · ImageNet | test error (%): 24.14 -> 24.32 | Hurt |
| set alpha to 2 instead of 4 (ablation) | FRAGrow ResNet · ImageNet | test error (%): 24.32 -> 24.86 | Hurt |
| set alpha to 6 instead of 4 (ablation) | FRAGrow ResNet · ImageNet | test error (%): 24.32 -> 24.27 | Helped |
| set alpha to 2 instead of 4 (ablation) | FRAGrow VGG · CIFAR100 | test error (%): 26.57 -> 26.75 | Hurt |
| replace layerwise growth order with circulation growth (ablation) | FRAGrow VGG · ImageNet | test error (%): 24.39 -> 24.18 | Helped |
| initialize new layers from historical ensemble weights (ablation) | FRAGrow · ImageNet | test error (%): ResNet 24.32 -> 24.65; VGG 24.39 -> 24.24 | Mixed |
Surprising findings
- Neural growth acts as a regularizer: faster growth (larger average training epochs) yields lower training error, and on overfitting datasets this regularization can slightly improve test error.
- Existing periodic and convergent growth policies cause accuracy drops of about 1.1% and 1.7% on underfitting ImageNet.
Limitations
- Saves less training time than periodic growth because growth is never slower than the periodic interval.
- Only one hyperparameter alpha, but results sensitive to it; evaluated only on image classification with CNNs.
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet TinyImageNet
Benchmarks: CIFAR10 CIFAR100 ImageNet
Metrics: test error normalized training time training error
Baselines: Periodic growth Convergent growth LipGrow Vanilla (large model from scratch) Small model from scratch
Builds on: periodic growth convergent growth LipGrow moment growth He initialization
Closest papers by meaning
- Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence (NeurIPS 2019 · similarity 0.87)
- When Will Gradient Regularization Be Harmful? (ICML 2024 · similarity 0.84)
- Masked Structural Growth for 2x Faster Language Model Pre-training (ICLR 2024 · similarity 0.84)
- Dynamic Neural Regeneration: Enhancing Deep Learning Generalization on Small Datasets (NeurIPS 2024 · similarity 0.83)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.83)
- Navigating Scaling Laws: Compute Optimality in Adaptive Model Training (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.