Research map / Architectures and efficiency / CNN design
Inference, Learning and Attention Mechanisms that Exploit and Preserve Sparsity in CNNs
Timo Häckel, Mikhail Usvyatsov, Silvano Galliani, Jan Dirk Wegner, Konrad Schindler B
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use sparse direct convolution instead of dense TensorFlow convolution | Convolution layer · Random tensors r=256, filter density 0.1 | speedup: 14x faster with density regularisation; 7x faster including k-selection | Improved |
| use sparse representation instead of dense at 512^3 | Convolution layer · Random tensors memory | memory (GB): 137.28 -> 0.8 | Improved |
| use sparse network instead of OctNet at 256^3 | Sparse network · Modelnet40 training memory | memory: OctNet out of memory; sparse 2880 MB | Improved |
| use sparse network instead of OctNet at low resolution 16^3 | Sparse network · Modelnet40 training time | training time per epoch: 16s -> 2m10s | Hurt |
| use sparse network instead of OctNet at 128^3 | Sparse network · Modelnet40 training time | training time per epoch: 34m43s -> 6m20s | Improved |
| loosen density upper bound from 0.035 to 0.075 (ablation) | Sparse net · Thresholded MNIST, 10 epochs | accuracy: 0.954 -> 0.976 | Improved |
| loosen density upper bound from 0.035 to 0.075 (ablation) | Sparse net · Thresholded MNIST, 60 epochs | accuracy: 0.985 -> 0.992 | Improved |
| add density regularisation and pruning (lambda=0.3) | Sparse net · Modelnet40 runtime after 90 epochs | runtime: 51% faster | Improved |
| add density regularisation and pruning (lambda=0.3) | Sparse net · Modelnet40 accuracy | accuracy | No clear effect |
| select k largest signed responses instead of largest absolute values (ablation) | k-selection attention · Modelnet40 16^3 | accuracy | Improved |
| tighten density upper bound from 1.0 to 0.06 (ablation) | Sparse net · Modelnet40 16^3 | accuracy: drop reaches approximately 3 percent points | Hurt |
| use sparse network instead of dense at 256^3 | Sparse network · ScanNet classification training speed | speedup: more than 10x faster | Improved |
Closest papers by meaning
- Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks (JMLR 2021 · similarity 0.92)
- On Implicit Filter Level Sparsity in Convolutional Neural Networks (CVPR 2019 · similarity 0.92)
- Accelerating Sparse Convolution with Column Vector-Wise Sparsity (NeurIPS 2022 · similarity 0.91)
- CNN Filter DB: An Empirical Investigation of Trained Convolutional Filters (CVPR 2022 · similarity 0.90)
- Faster CNNs with Direct Sparse Convolutions and Guided Pruning (ICLR 2017 · similarity 0.90)
- Inducing and Exploiting Activation Sparsity for Fast Inference on Deep Neural Networks (ICML 2020 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
