Research map / Architectures and efficiency / Pruning and efficient tuning
UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UPDP, a depth (block) pruning framework for CNNs and vision transformers that converts blocks into reparameterizable mergeable forms and progressively transitions them to pruned subnets, evaluated on ImageNet-1K.
Problem
Channel pruning handles depthwise/inverted-residual efficient CNNs poorly, and existing depth pruners (DepthShrinker, Layer-Folding) cannot handle LayerNorm/GroupNorm models or ViTs and damage baseline weights by hard-removing activations.
Approach
Replace activations with identity and LN/GN with BN inside blocks (plus an inserted activation+BN at block end) so that the block becomes mergeable via reparameterization; a supernet with sandwich-rule training and genetic subnet search selects which blocks to prune, and a progressive lambda-blend schedule gradually transitions baseline blocks to pruned blocks before merging.
Key results
- ConvNeXtV1-T-UPDP-P3 reaches 81.9% top-1 at 3.3x speedup, versus 81.8% for DeiT-B at 1.5x and 81.1% for Swin-T.
- On DeiT-Tiny, Ours-P6 reaches 70.3% top-1 at 1.26x speedup with a 1.9% drop from 72.2% baseline, outperforming S2ViTE and WD-Pruning on accuracy.
- On ResNet34, the P10 pruned model reaches 71.8% (vs 70.8% for US and 71.6% for MetaPruning at 0.5x) at 1.43x speedup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Choose the higher-accuracy subnet before finetuning (ablation) | ResNet34-P10 subnet search · ImageNet-1K validation | Acc1 (%) after finetuning: 71.2 -> 71.8 | Helped |
| Use progressive rather than direct subnet training (ablation) | ResNet34-P10 · ImageNet-1K validation | Acc1 (%): 71.3 -> 71.8 | Helped |
| Use progressive rather than direct subnet training (ablation) | MBV2-1.4-P9 · ImageNet-1K validation | Acc1 (%): 73.1 -> 73.8 | Helped |
| Use progressive rather than direct subnet training (ablation) | ConvNeXtV1-T-P3 · ImageNet-1K validation | Acc1 (%): 81.6 -> 81.9 | Helped |
| Use progressive rather than direct subnet training (ablation) | DeiT-Tiny-P6 · ImageNet-1K validation | Acc1 (%): 69.5 -> 70.3 | Helped |
| Apply UPDP instead of DepthShrinker’s DS-C subnet training | MobileNetV2-1.4-P9 · ImageNet-1K validation | Acc1 (%): 73.1 -> 73.8 | Helped |
| Apply UPDP to DeiT-Tiny with 6 layers pruned | DeiT-Tiny · ImageNet-1K validation | speedup: 1.26 | Helped |
Surprising findings
- Progressive training improves accuracy only modestly (0.3%-0.8%) over direct activation removal.
Limitations
- Evaluated only on ImageNet-1K classification and a limited set of models.
- Gains over direct activation removal are modest; ViT results lag some pruning methods on speedup-accuracy trade-off details.
- Kernel size and stride changes are needed to reduce merging error, adding hand-tuned steps.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K
Metrics: top-1 accuracy FLOPs speedup params
Baselines: MetaPruning Universally Slimmable Networks DepthShrinker S2ViTE WD-Pruning XPruner SCOP HVT ConvNeXtV2 EfficientFormerV2
Builds on: DepthShrinker Layer-Folding DBB RepVGG BigNAS
Closest papers by meaning
- Width & Depth Pruning for Vision Transformers (AAAI 2022 · similarity 0.90)
- Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework (ICML 2021 · similarity 0.89)
- Structured Pruning of Neural Networks With Budget-Aware Regularization (CVPR 2019 · similarity 0.89)
- Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective (ICML 2026 · similarity 0.89)
- MDP: Multidimensional Vision Model Pruning with Latency Constraint (CVPR 2025 · similarity 0.88)
- Preserving Deep Representations in One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.