Research map / Architectures and efficiency / RNNs and normalization
Inducing and Exploiting Activation Sparsity for Fast Inference on Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Square Hoyer activation regularization instead of L1 (ablation) | ResNet18, ResNet50, Mobilenet activations · ImageNet | average activation sparsity: ResNet18 55%->62%; ResNet50 54%->61%; Mobilenet 52%->58% | Helped |
| Dynamic FATReLU thresholding added on top of Hoyer regularization (ablation) | Hoyer-regularized models · ImageNet | average activation sparsity: ResNet18 62%->67%; ResNet50 61%->65%; Mobilenet 58%->60% | Helped |
| Hoyer regularization plus dynamic thresholding with sparse-input convolution | ResNet50 / 2x Wide ResNet50 / ResNet101 / 2x Wide ResNet101 · ImageNet, own dense baseline | end-to-end speedup: 1.53x to 2.57x | Helped |
| Sparse-input convolution with Hoyer + boosting vs MXNet+MKL-DNN dense CPU inference | ResNet18, ResNet50, Mobilenet · ImageNet, batch 64, Intel CPU | inference time (ms): ResNet18 113.41->60.92 ms; ResNet50 317.49->180.5 ms; Mobilenet 88.55->49.77 ms | Helped |
| Channel-wise thresholding instead of layer-wise dynamic thresholding (ablation) | ResNet18 activations · ImageNet | average activation sparsity: only ~2% extra sparsity over dynamic thresholding at significantly increased cost | Hurt |
| Static FATReLU threshold boosting without retraining (ablation) | ResNet18 activations · ImageNet | average activation sparsity: under 3% globally | No clear effect |
| Hoyer regularization with dynamic thresholding also keeps accuracy loss near zero | ResNet/Mobilenet activation-sparse models · ImageNet | top-1 accuracy: < 0.3% accuracy loss | No clear effect |
Closest papers by meaning
- DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.91)
- Faster CNNs with Direct Sparse Convolutions and Guided Pruning (ICLR 2017 · similarity 0.90)
- Accelerating Convolutional Neural Networks via Activation Map Compression (CVPR 2019 · similarity 0.90)
- Exploring the Benefit of Activation Sparsity in Pre-training (ICML 2024 · similarity 0.89)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.88)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.