Research map / Architectures and efficiency / Pruning and efficient tuning
DEPrune: Depth-wise Separable Convolution Pruning for Maximizing GPU Parallelism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DEPrune, a GPU-aware pruning method for depth-wise separable convolutions that applies fine-grained weight pruning to depth-wise convs after diagonal-wise refactorization (yielding structured zero columns), filter pruning to point-wise convs, plus balanced workload tuning and hardware-aware sparsity recalibration to speed up DSConv inference.
Problem
Pruning compact depth-wise separable CNNs is hard: depth-wise convolutions have very few parameters, so coarse pruning hurts accuracy, and fine-grained pruning gives little GPU speedup because DW-conv underutilizes GPU parallelism.
Approach
Apply magnitude-based fine-grained pruning to DW-conv weights after diagonal-wise refactorization so that pruned weights form zero columns (structured sparsity usable by GEMM kernels), combine with filter pruning on PW-conv2 only, and balance per-sub-GEMM pruning ratios and align pruned vectors to GPU tile size of 32.
Key results
- DEPrune-BH reaches 6.15x DW-conv speedup and 3.74x total speedup on EfficientNet-B0 at -0.8% top-1 (77.6% baseline) on ImageNet.
- DEPrune-BH on MobileNet-V3-Small reports 4.12x total speedup with -0.6% top-1 accuracy drop.
- Balanced workload tuning (DEPrune-B) raises MobileNet-V2 DW-conv speedup from 2.75x to 3.35x with only -0.41% accuracy change.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add BWT to DEPrune (ablation) | DEPrune · ImageNet, MobileNet-V2 | DW-conv speedup: 2.75× → 3.35× | Helped |
| Add BWT to DEPrune (ablation) | DEPrune · ImageNet, MobileNet-V3-Small | DW-conv speedup: 3.88× → 5.09× | Helped |
| Add BWT to DEPrune (ablation) | DEPrune · ImageNet, EfficientNet-B0 | DW-conv speedup: 4.51× → 5.79× | Helped |
| Add BWT to DCP (ablation) | DCP · CIFAR-10, EfficientNet-B0 | accuracy: 91.16% → 91.33% | Helped |
| Add HSR to DEPrune-B (ablation) | DEPrune-B · ImageNet, EfficientNet-B0 | DW-conv speedup: 5.79× → 6.15× | Helped |
| Add HSR to DEPrune-B (ablation) | DEPrune-B · ImageNet, MobileNet-V3-Small | Top-1 accuracy: 67.17 → 67.18 | No clear effect |
| Use DEPrune instead of channel pruning (ablation) | MobileNet-V2 · CIFAR-10, 50% DW-conv and 30% PW-conv pruning | accuracy: 93.07% → 93.30% | Helped |
| Use DEPrune-B instead of NVIDIA n:m sparsity (ablation) | MobileNet-V2 · CIFAR-10, 50% pruning | accuracy: 92.99% → 93.30% | Helped |
| Apply BCBP rather than ℓ2-norm filter pruning to PW-conv (ablation) | DEPrune-BH · ImageNet, MobileNet-V2 | Top-1 accuracy: 71.54 → 71.23 | Hurt |
| Apply BCBP rather than ℓ2-norm filter pruning to PW-conv (ablation) | DEPrune-BH · ImageNet, MobileNet-V3-Small | Top-1 accuracy: 67.17 → 66.09 | Hurt |
| Apply BCBP rather than ℓ2-norm filter pruning to PW-conv (ablation) | DEPrune-BH · ImageNet, EfficientNet-B0 | Top-1 accuracy: 76.83 → 75.91 | Hurt |
| Prune DW-conv with DCP rather than channel pruning (ablation) | MobileNet-V2 · ImageNet, 40% pruning | Top-1 accuracy drop: 1.66% versus 0.94% | Helped |
Surprising findings
- Channel pruning on DW-conv loses representation power even at 40% PR, while fine-grained DCP keeps accuracy at 50-70% PR.
- Filter pruning of PW-conv2 indirectly removes DW-conv and PW-conv1 channels, so DEPrune restricts PW-conv pruning to PW-conv2 only.
Limitations
- Method is specialized for depth-wise convolution and hard to apply to Vision Transformers.
- Relies on GPU tile size (32) and custom GEMM kernels; hardware-specific.
- Evaluation limited to ImageNet and CIFAR-10 image classification with three CNN families.
Experimental setup
Datasets: ImageNet CIFAR-10
Benchmarks: ImageNet CIFAR-10
Metrics: Top-1 accuracy inference time speedup FLOPs reduction peak memory usage
Baselines: CafeNet AMC GFS FPGM RLAL MetaPruning BCBP NVIDIA n:m sparsity
Builds on: Diagonal-wise Refactorization magnitude pruning filter pruning (FP) DepthShrinker
Closest papers by meaning
- Balanced Column-Wise Block Pruning for Maximizing GPU Parallelism (AAAI 2023 · similarity 0.89)
- UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer (AAAI 2024 · similarity 0.87)
- MDP: Multidimensional Vision Model Pruning with Latency Constraint (CVPR 2025 · similarity 0.85)
- Resource-Efficient Transformer Pruning for Finetuning of Large Models (CVPR 2024 · similarity 0.85)
- Preserving Deep Representations in One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework (ICLR 2025 · similarity 0.85)
- REPrune: Channel Pruning via Kernel Representative Selection (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.