Research map / Architectures and efficiency / Pruning and efficient tuning
Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Auto-Train-Once (ATO), an end-to-end structural pruning method in which a GRU-based controller network generates group masks over zero-invariant groups during training, with a stochastic mirror-descent-style optimizer and convergence analysis, removing the need for fine-tuning.
Problem
Structural pruning of DNNs usually needs multi-stage pipelines (pretrain, identify redundancy, fine-tune), and prior end-to-end methods (OTO, OTOv2) use static pruning-group selection that can converge to poor local optima and rely on strong convergence assumptions.
Approach
A controller network (bi-directional GRU with Gumbel-Sigmoid and straight-through estimator) dynamically produces binary masks over zero-invariant groups during training, with group-specific regularization set by the mask and a FLOPs regularizer, trained alternately with the target model on a small 5% subset; pruned structures are removed after training without fine-tuning.
Key results
- On CIFAR-10 ResNet-18, ATO reaches +0.10% accuracy change at 79.8% pruned FLOPs versus OTOv2's -0.16% at 79.7%.
- On ImageNet ResNet-50 at 61.7% pruned FLOPs, ATO achieves 76.07% top-1, reported as 0.87% better than OTOv2 at similar FLOPs.
- On ImageNet ResNet-34, ATO reaches 72.92% top-1 (-0.39%) at 44.1% pruned FLOPs, outperforming SCOP and DMC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a controller network to dynamically select pruning masks instead of static pruning groups | ATO vs. OTOv2 · CIFAR-10, ResNet-18 | pruned accuracy: 92.86% -> 94.51% | Helped |
| Use a controller network to dynamically select pruning masks | ATO vs. OTOv2 · ImageNet, ResNet-50 | Top-1 accuracy: 75.20% -> 76.07% | Helped |
| Use a controller network to dynamically select pruning masks | ATO vs. OTOv2 · CIFAR-100, ResNet-34 | pruned accuracy: 76.31% -> 78.54 % | Helped |
| Set the regularization coefficient λ too small (ablation) | ATO · CIFAR-10, ResNet-56 | test accuracy | Hurt |
| Vary the FLOPs regularization strength γ (ablation) | ATO · CIFAR-10, ResNet-56 | test accuracy | No clear effect |
| Set γ too small (ablation) | ATO controller network · CIFAR-10, ResNet-56 | FLOPs target attainment | Hurt |
| Vary the warm-up steps Tw (ablation) | ATO · CIFAR-10, ResNet-56 | test accuracy | No clear effect |
| Use the proximal gradient projector instead of the Half-Space projector (ablation) | ATO · CIFAR-10, ResNet-56 | test accuracy | No clear effect |
Surprising findings
- The paper reports no meaningful difference between the proximal gradient projector and the half-space projector, suggesting the controller network rather than the projection operator drives performance.
Limitations
- Convergence analysis relies on assumptions that are only partly stated (assumption references are broken in the text).
- Results on CIFAR-100 with ResNet-18 show a -0.07% accuracy change, and ResNet-50 ImageNet at 55.2% is slightly worse than some baselines in its own comparisons.
- Hyperparameters (lambda, gamma, Tstart, Tend, Tw) require tuning per model and dataset, with details deferred to supplementary material.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet
Benchmarks: CIFAR-10 CIFAR-100 ImageNet
Metrics: top-1 accuracy top-5 accuracy accuracy change (ΔAcc) pruned FLOPs
Baselines: OTOv2 OTOv1 DCP SCOP DMC GNN-RL FPGM DTP
Builds on: OTO OTOv2 DHSPG Gumbel-Softmax
Closest papers by meaning
- Only Train Once: A One-Shot Neural Network Training And Pruning Framework (NeurIPS 2021 · similarity 0.91)
- AutoPrune: Automatic Network Pruning by Regularizing Auxiliary Parameters (NeurIPS 2019 · similarity 0.90)
- Learning Pruning-Friendly Networks via Frank-Wolfe: One-Shot, Any-Sparsity, And No Retraining (ICLR 2022 · similarity 0.88)
- Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon (NeurIPS 2017 · similarity 0.88)
- AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates (AAAI 2020 · similarity 0.88)
- Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.