Research map / Architectures and efficiency / RNNs and normalization
SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks at the Edge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SparseProp for backpropagation on unstructured sparse weights | Linear layers · Linear layer backward, single-core, 99% sparsity | speedup: 5× | Helped |
| Use SparseProp for backpropagation on unstructured sparse weights | Convolutional layers with small inputs · Small input, single-core, 99% sparsity | speedup: 19x | Helped |
| Vectorize the sparse convolution kernel over ON | Convolutional layers with large inputs · Large input, backward pass | speedup: 9x | Helped |
| Increase uniform pruning from 90% to 97% | ResNet50 sparse transfer · Transfer tasks, single-core | end-to-end speedup: 1.23× -> 1.76× | Helped |
| Use uniform 97% pruning instead of global 95% pruning | ResNet50 sparse transfer · Transfer tasks, single-core | end-to-end speedup: 1.27× -> 1.76× | Helped |
| Use a 95% uniformly pruned ResNet18 rather than the dense model | ResNet18 transfer learning · 12 target datasets | accuracy: 2.85% average drop; maximum gap of 4.5% Top-1 | Hurt |
| Use a 95% uniformly pruned ResNet18 with SparseProp | ResNet18 transfer learning · 12 target datasets | end-to-end speedup: 1.58x | Helped |
| Increase uniform sparsity from 90% to 97% | ResNet18 from-scratch training · Specialized vision tasks | end-to-end speedup: 1.17× -> 1.41× | Helped |
| Increase uniform sparsity from 90% to 97% | BERT-base from-scratch fine-tuning · QQP | end-to-end speedup: 1.04× -> 1.31× | Helped |
| Increase uniform sparsity from 90% to 97% | BERT-base from-scratch fine-tuning · QQP | accuracy: 90.09 -> 87.81 | Hurt |
| Use global rather than uniform pruning at 95% sparsity | ResNet18 from-scratch training · Specialized vision tasks | accuracy: Global-95% shows similar accuracy to Uniform-90% | Helped |
| Permute small-input convolution tensors to vectorize over the batch | Sparse convolution kernel · Small input tensors | runtime: up to 5x | Helped |
Closest papers by meaning
- meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting (ICML 2017 · similarity 0.91)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.89)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.88)
- Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once (ICML 2024 · similarity 0.88)
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.88)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.