Research map / Architectures and efficiency / CNN design
LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LayerMerge, which jointly prunes convolution and activation layers via a surrogate knapsack-style objective solved exactly with dynamic programming, avoiding the kernel-size blow-up of depth compression.
Problem
Depth compression of CNNs merges consecutive convolutions, which enlarges merged kernel sizes and undermines the latency reduction from fewer layers.
Approach
Jointly select which activation layers and which convolution layers to remove (replacing removed convs with 1x1 identity depthwise convs so kernel size does not grow), using a surrogate objective over lookup tables of latency and importance indexed by merged kernel size, solved exactly by dynamic programming.
Key results
- On ResNet-34 (ImageNet), LayerMerge-71% reaches 74.26% accuracy at 1.36x speed-up versus 73.49% for Depth-78% at 1.24x.
- On MobileNetV2-1.0, LayerMerge-55% keeps 72.99% accuracy (no loss vs 72.89% original) at 1.63x speed-up, versus 72.83% for Depth-74% at 1.62x.
- On DDPM (CIFAR10), LayerMerge-73% achieves FID 4.16 at 1.13x speed-up, versus FID 4.21 for Depth-89% at 1.04x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly optimize activation-layer removal and convolution-layer pruning instead of Depth-78% | ResNet-34 · ImageNet val | top-1 accuracy (%): 73.49 -> 74.26 | Helped |
| jointly optimize activation-layer removal and convolution-layer pruning instead of Depth-74% | MobileNetV2-1.0 · ImageNet val | top-1 accuracy (%): 72.83 -> 72.99 | Helped |
| jointly optimize activation-layer removal and convolution-layer pruning instead of pruning convolution layers only | MobileNetV2-1.0 · ImageNet val | top-1 accuracy (%): 69.66 -> 72.99 | Helped |
| jointly optimize activation-layer removal and convolution-layer pruning instead of Depth-62% | MobileNetV2-1.4 · ImageNet val | top-1 accuracy (%): 74.68 -> 74.91 | Helped |
| jointly optimize activation-layer removal and convolution-layer pruning instead of Depth-89% | DDPM · CIFAR10 val | FID: 4.21 -> 4.16 | Helped |
| apply LayerMerge after Diff-Pruning instead of applying Depth after Diff-Pruning | DDPM · CIFAR10 val | FID: 9.09 -> 8.92 | Helped |
| jointly optimize activation and convolution pruning instead of sequentially applying Depth then LayerOnly (ablation) | MobileNetV2-1.0 · ImageNet val | top-1 accuracy (%): 71.72 -> 71.89 | Helped |
| jointly optimize activation and convolution pruning instead of sequentially applying Depth then LayerOnly (ablation) | MobileNetV2-1.0 · ImageNet val | top-1 accuracy (%): 70.14 -> 70.81 | Helped |
| fine-tune LayerMerge-55% for fewer epochs (180 to 90) (ablation) | MobileNetV2-1.0 · ImageNet val | top-1 accuracy (%): 72.99 -> 72.73 | Hurt |
| fine-tune LayerMerge-55% for fewer epochs (180 to 30) (ablation) | MobileNetV2-1.0 · ImageNet val | top-1 accuracy (%): 72.99 -> 72.06 | Hurt |
| fine-tune LayerMerge-55% for fewer epochs (180 to 20) (ablation) | MobileNetV2-1.0 · ImageNet val | top-1 accuracy (%): 72.99 -> 71.59 | Hurt |
| apply knowledge distillation during fine-tuning (ablation) | LayerMerge on MobileNetV2-1.0 · ImageNet val | top-1 accuracy (%): 72.99 -> 73.14 | Helped |
Surprising findings
- Diff-Pruning channel pruning outperforms LayerMerge on DDPM, suggesting DDPM has more channel redundancy than other models.
- Sequentially applying Depth then LayerOnly underperforms joint optimization, and LayerOnly alone degrades MobileNetV2 accuracy substantially (e.g., 69.66% at 1.30x).
Limitations
- Relies on a surrogate objective that sums per-layer importance and latency, which approximates the true merged-network objective.
- Requires precomputing importance and latency lookup tables on the target device, and fine-tuning after pruning.
- Gains are smaller on channel-redundant architectures like DDPM, where channel pruning is stronger.
Experimental setup
Datasets: ImageNet CIFAR10
Benchmarks: ImageNet CIFAR10
Metrics: top-1 accuracy FID latency speed-up (PyTorch and TensorRT)
Baselines: Depth (depth compression) LayerOnly HALP AMC MetaPruning Diff-Pruning Knowledge distillation
Builds on: Depth compression (Kim et al. 2023 / DepthShrinker-style) HALP Diff-Pruning DDPM
Closest papers by meaning
- Convolutional Neural Network Pruning With Structural Redundancy Reduction (CVPR 2021 · similarity 0.88)
- Efficient Latency-Aware CNN Depth Compression via Two-Stage Dynamic Programming (ICML 2023 · similarity 0.88)
- Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition (NeurIPS 2021 · similarity 0.88)
- “Learning-Compression” Algorithms for Neural Net Pruning (CVPR 2018 · similarity 0.88)
- Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions (NeurIPS 2022 · similarity 0.88)
- Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework (ICML 2021 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.