Research map / Architectures and efficiency / Pruning and efficient tuning
Network Expansion for Practical Training Acceleration
Ning Ding, Yehui Tang, Kai Han, Chao Xu, Yunhe Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a narrower ResNet32 first, then progressively add channels | ResNet32 · CIFAR10 | Top1 Acc.(%): 94.41 -> 94.36 | No clear effect |
| Train a narrower ResNet56 first, then progressively add channels | ResNet56 · CIFAR100 | Top1 Acc.(%): 75.80 -> 75.76 | No clear effect |
| Train a narrower network first, then progressively add channels | ResNet101 · ImageNet-1k | Top1 Acc.(%): 77.4 -> 77.2 | No clear effect |
| Train a shallow model and expand depth using EMA-model layers | DeiT-base · ImageNet-1k | Top1 Acc.(%): 81.8 -> 81.5 | No clear effect |
| Expand the ResNet101 backbone from a narrower starting width | DeepLabV3+ with ResNet101 · Cityscapes | mIoU(%): 76.9 -> 75.1 | Hurt |
| Set α0 = 0.5 and ng = 2 instead of α0 = 0.5 and ng = 3 (ablation) | DeepLabV3+ with ResNet101 · Cityscapes | mIoU(%): 75.1 -> 76.4 | Improved |
| Increase the initial width and reduce the number of expansion stages (ablation) | ResNet56 · CIFAR100 | Top1 Acc.: 75.76 -> 75.91; 1.45× -> 1.35× | Mixed |
| Reduce the starting width to α0 = 0.125 (ablation) | ResNet56 · CIFAR100 | Top1 Acc.: 75.76 -> 75.31 | Hurt |
| Use α0 = 1/3 and expand in 3 stages rather than α0 = 0.5 and 2 stages (ablation) | DeiT-base · ImageNet-1k | Top1 Acc.: 81.5 -> 80.2 | Hurt |
| Initialize added filters with orthogonal weights instead of random weights (ablation) | ResNet56 · CIFAR100 | Top1 Acc.: 75.39 -> 75.76 | Improved |
| Initialize added filters with orthogonal weights instead of replicated old filters (ablation) | ResNet56 · CIFAR100 | Top1 Acc.: 74.67 -> 75.76 | Improved |
Closest papers by meaning
- Growing Efficient Deep Networks by Structured Continuous Sparsification (ICLR 2021 · similarity 0.88)
- Deep Expander Networks: Efficient Deep Networks from Graph Theory (ECCV 2018 · similarity 0.88)
- ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks (NeurIPS 2020 · similarity 0.87)
- Accelerating CNN Training by Pruning Activation Gradients (ECCV 2020 · similarity 0.87)
- Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective (AAAI 2025 · similarity 0.86)
- Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
