Research map / Architectures and efficiency / Pruning and efficient tuning
Entropy Induced Pruning Framework for Convolutional Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AFIE, an SVD-eigenvalue entropy score for ranking filters and allocating per-layer pruning ratios, so structured pruning can be done from a barely trained (1-epoch) CNN with stable scores.
Problem
Structured pruning of CNNs for image classification usually requires a fully trained original model; pruning results degrade when the model is under-trained, and the process is time-consuming.
Approach
Fold each conv layer's weight tensor into a 2D matrix, take SVD eigenvalues, normalize them (0-1 scaling then softmax), and compute Shannon entropy as Average Filter Information Entropy (AFIE); lower entropy implies lower rank and more redundancy, so layers get higher pruning ratios. Claims AFIE scores are nearly identical for 1-epoch and fully trained models.
Key results
- On AlexNet/MNIST with 1-epoch training, AFIE pruning reaches 0.9902 top-1 accuracy with 0.83M parameters (vs. 3.52M original), outperforming listed baselines at similar size.
- On VGG-16/CIFAR-10 with 1-epoch training, AFIE reaches 0.9312 accuracy at 19.15M parameters and 1.5e8 FLOPs, above the compared methods in that setting.
- On ResNet-50/ImageNet with 1-epoch training, AFIE achieves 0.745 top-1 accuracy at 21.88M parameters and 3.70e9 FLOPs, versus 0.758 when fully trained.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply AFIE-based pruning to AlexNet | AlexNet · MNIST | Top1-Acc: 0.9891 -> 0.9902 | Helped |
| apply AFIE-based pruning to VGG-16 | VGG-16 · CIFAR-10 | Top1-Acc: 0.9293 -> 0.9312 | Helped |
| apply AFIE-based pruning to ResNet-50 | ResNet-50 · ImageNet | Top1-Acc: 0.743 -> 0.745 | Helped |
| apply AFIE-based pruning instead of Taylor pruning | AlexNet · MNIST | Top1-Acc: 0.9925 -> 0.9920 | Hurt |
| apply AFIE-based pruning instead of DCP | VGG-16 · CIFAR-10 | Top1-Acc: 0.9335 -> 0.9335 | No clear effect |
| apply AFIE-based pruning instead of DCP | ResNet-50 · ImageNet | Top1-Acc: 0.758 -> 0.758 | No clear effect |
| train the original AlexNet for 20 epochs instead of 1 epoch (ablation) | AFIE-pruned AlexNet · MNIST | Top1-Acc: 0.9902 -> 0.9920 | Helped |
| train the original VGG-16 for 150 epochs instead of 1 epoch (ablation) | AFIE-pruned VGG-16 · CIFAR-10 | Top1-Acc: 0.9312 -> 0.9335 | Helped |
| train the original ResNet-50 for 150 epochs instead of 1 epoch (ablation) | AFIE-pruned ResNet-50 · ImageNet | Top1-Acc: 0.745 -> 0.758 | Helped |
| train the original model for more epochs before pruning (ablation) | AFIE · AlexNet on MNIST; VGG-16 on CIFAR-10; ResNet-50 on ImageNet | AFIE: 0.015 -> 0.015; 0.016 -> 0.016; 0.064 -> 0.064 | No clear effect |
Surprising findings
- AFIE scores per layer are reported as identical whether the model was trained for 1 epoch or fully trained (Tables 1-3).
Limitations
- Filters within each layer are assumed equally important, and the pruned filters are chosen randomly within each layer.
- Accuracy drops of up to about 1% occur when the original model is trained for only one epoch, and the pruned models are fully retrained afterward.
- Theoretical justification for stability (Lipschitz argument) is only sketched and relegated to an appendix not included in the provided text.
Experimental setup
Datasets: MNIST CIFAR-10 ImageNet
Benchmarks: MNIST CIFAR-10 ImageNet
Metrics: top-1 accuracy parameters FLOPs AFIE score stability
Baselines: ThiNet DCP SEP l1-norm pruning Network Slimming Taylor pruning
Builds on: Network Slimming ThiNet DCP
Closest papers by meaning
- Provable Filter Pruning for Efficient Neural Networks (ICLR 2020 · similarity 0.90)
- Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration (CVPR 2020 · similarity 0.90)
- Rethinking the Pruning Criteria for Convolutional Neural Network (NeurIPS 2021 · similarity 0.89)
- DPFPS: Dynamic and Progressive Filter Pruning for Compressing Convolutional Neural Networks from Scratch (AAAI 2021 · similarity 0.89)
- Auto-Balanced Filter Pruning for Efficient Convolutional Neural Networks (AAAI 2018 · similarity 0.89)
- Interspace Pruning: Using Adaptive Filter Representations To Improve Training of Sparse CNNs (CVPR 2022 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.