Research map / Architectures and efficiency / RNNs and normalization
DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace inverted residual blocks with merged dense convolutions (DepthShrinker block merging) | MobileNetV2 / ResNet families · ImageNet, real-device throughput | FPS: up to 3.91x-4.38x on V100/2080Ti for MobileNetV2 with same FLOPs | Helped |
| Pretraining the merged network instead of training from scratch (ablation) | MBV2-DS variants · ImageNet | Top-1 accuracy: +0.42% to +1.27% | Helped |
| Add free ReLU6 after each merged convolution during fine-tuning (ablation) | MBV2-1.4-DS variants · ImageNet | Top-1 accuracy: up to +2.08% (DS-E); up to 0.81-2.08 gains in Table 4 | Helped |
| Learned differentiable search for which activation functions to remove vs. random search (ablation) | MBV2-1.4-DS-E and MBV2-DS-D · ImageNet / RTX 2080Ti | Top-1 accuracy and throughput: 4.30%/5.46% higher accuracy with 1.24x/1.19x throughput over random search | Helped |
| Hard layer-wise pruning of the same blocks instead of merging (ablation) | MobileNetV2-1.4 · ImageNet / RTX 2080Ti | Top-1 accuracy and FPS: DepthShrinker gives 2.80% higher accuracy with 1.50x throughput than smallest layer-pruned model | Hurt |
| Expand-then-shrink training (DepthShrinker+) | VGG11, VGG13, MCUNet, MobileNetV2-0.5, MobileNetV2 · ImageNet | Top-1 accuracy: +1.26% to +1.62% over standard training | Helped |
| Self-distillation during fine-tuning and longer training (360 epochs) (ablation) | MBV2-1.4-DS and MBV2-DS variants · ImageNet | Top-1 accuracy: accuracy boosted by up to 2.57% | Helped |
| Replace MetaPruning-0.35x with DepthShrinker-derived model at similar accuracy scale | MobileNetV2-1.4 DS-F · ImageNet / Tesla V100 | FPS and accuracy: 3.06% higher accuracy with 1.53x throughput over MetaPruning-0.35x | Helped |
| Shrinking blocks on CPU devices (merged dense convs) | MBV2-DS and MBV2-1.4-DS variants · Pixel 3 and Raspi 4 latency | latency (s): latency reduced by up to 38.4% (Pixel 3) and 43.1% (Raspi 4) | Helped |
Closest papers by meaning
- "Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach (NeurIPS 2022 · similarity 0.90)
- Fast Sparse ConvNets (CVPR 2020 · similarity 0.88)
- Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition (NeurIPS 2021 · similarity 0.88)
- Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks (ICLR 2017 · similarity 0.88)
- HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks (AAAI 2023 · similarity 0.88)
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.