Research map / Architectures and efficiency / Pruning and efficient tuning
BilevelPruning: Unified Dynamic and Static Channel Pruning for Convolutional Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UDSP, which unifies static and dynamic channel pruning of CNNs via bi-level optimization so that the static sub-network is learned implicitly through dynamic sub-networks, then removes redundant static channels and finetunes with dynamic flexibility.
Problem
Dynamic channel pruning keeps all weights, incurring storage costs, while static pruning fixes a single sub-network and limits capacity; existing methods treat the two separately with hand-designed fusion.
Approach
Formulates static and dynamic channel pruning as a bi-level optimization: the outer problem learns dynamic gates on the best static sub-network (approximated by one-step lookahead), with an efficient Jacobian-product approximation for the second-order term, so both are learned end-to-end without handcrafted fusion.
Key results
- On CIFAR-10 ResNet-56, UDSP reaches 93.78% accuracy (+0.66% vs base) with 50.1% FLOPs and 20.0% parameter reduction, beating SEP (+0.32%).
- On ImageNet ResNet-50, UDSP achieves 76.51% top-1 (+0.38%) with 58.4% FLOPs reduction, outperforming DepGraph and DTP.
- On ImageNet ResNet-18, UDSP achieves 69.48% top-1 (-0.28%) with 50.2% FLOPs reduction and 20.0% parameter reduction, above SEP (68.73%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine static and dynamic pruning with UDSP | CifarNet · CIFAR-10 | accuracy: 91.42% -> 91.89% | Helped |
| combine static and dynamic pruning with UDSP | ResNet-56 · CIFAR-10 | accuracy: 93.44% -> 93.78% | Helped |
| combine static and dynamic pruning with UDSP | ResNet-18 · ImageNet | Top-1 accuracy: 68.73% -> 69.48% | Helped |
| combine static and dynamic pruning with UDSP | ResNet-34 · ImageNet | Top-1 accuracy: 72.83% -> 72.91% | Helped |
| combine static and dynamic pruning with UDSP | ResNet-50 · ImageNet | Top-1 accuracy: 76.42% -> 76.51% | Helped |
| combine static and dynamic pruning with UDSP | MobileNet-V2 · ImageNet | Top-1 accuracy: 71.20% -> 71.72% | Helped |
| use bi-level optimization instead of iterative training (ablation) | UDSP on ResNet-56 · CIFAR-10 | accuracy: 93.55% -> 93.78% | Helped |
| prune 40.0% rather than 20.0% of parameters (ablation) | UDSP on ResNet-56 · CIFAR-10 | accuracy: 93.78% -> 93.32% | Hurt |
| prune 30.0% rather than 20.0% of parameters (ablation) | UDSP on ResNet-56 · CIFAR-10 | accuracy: 93.78% -> 93.64% | Hurt |
| increase parameter pruning from 20.0% to 30.0% (ablation) | UDSP on ResNet-56 · CIFAR-10 | parameter reduction: 20.0% -> 30.0% | Helped |
| increase parameter pruning from 20.0% to 40.0% (ablation) | UDSP on ResNet-56 · CIFAR-10 | parameter reduction: 20.0% -> 40.0% | Helped |
Surprising findings
- Naively optimizing static and dynamic pruning jointly or iteratively performs worse than the bi-level formulation (Fig. 2, Tab. 3).
- Pruning 40% of parameters after finetuning drops performance substantially, limiting dynamic flexibility to the level of static pruning.
Limitations
- Parameter pruning beyond roughly 30% degrades accuracy considerably, restricting dynamic flexibility.
- Reliance on one-step approximation of the inner static problem, which is approximate rather than exact.
Experimental setup
Datasets: CIFAR-10 ImageNet
Benchmarks: CIFAR-10 ImageNet
Metrics: top-1 accuracy accuracy change (delta-Acc) FLOPs reduction parameter reduction
Baselines: FBS SEP AMC FPGM HRank DSA CGNN FTWT IE SCOP GFP 3DP
Builds on: Feature boosting and suppression (FBS) Storage efficient dynamic pruning (SEP) DARTS Gumbel-sigmoid straight-through estimator
Closest papers by meaning
- Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers (ICLR 2018 · similarity 0.91)
- Dynamic Structure Pruning for Compressing CNNs (AAAI 2023 · similarity 0.90)
- Rethinking the Pruning Criteria for Convolutional Neural Network (NeurIPS 2021 · similarity 0.89)
- Dynamic Channel Pruning: Feature Boosting and Suppression (ICLR 2019 · similarity 0.89)
- Pruning from Scratch (AAAI 2020 · similarity 0.89)
- REPrune: Channel Pruning via Kernel Representative Selection (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.