Research map / Optimization / Optimizers
Efficient Neural Network Training via Forward and Backward Propagation Sparsification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| completely sparse forward and backward passes with VR-PGE instead of chain-rule estimators (vs GrowEfficient) | Sparse channel training · CIFAR-10 VGG-16 | train-cost savings (x): 1.22 -> 8.69 | Helped |
| completely sparse training vs GrowEfficient on WRN-28-10 | Sparse channel training · CIFAR-10 WRN-28-10 | Top-1 accuracy (%): 95.3 -> 95.6 | Helped |
| completely sparse training vs GrowEfficient on ResNet-50 | Sparse channel training · ImageNet-1K ResNet-50 | Top-1 accuracy (%): 75.2 -> 76.0 | Helped |
| push to extreme sparsity (10.8% params) on ResNet-50 | Sparse channel training · ImageNet-1K ResNet-50 | Top-1 accuracy (%): 76.0 -> 69.3 | Hurt |
| increase sparsity of VGG-19 on CIFAR-10 (wall-clock time) | Sparse channel training · CIFAR-10 VGG-19 | Top-1 accuracy (%): 93.84 -> 87.97 | Hurt |
| replace PGE with variance-reduced PGE (two-sample difference plus preconditioner) (ablation) | Policy gradient estimator for structure parameters · VGG-19 CIFAR-10 gradient variance | gradient variance: up to 3 orders of magnitude lower | Helped |
| sparse training wall-clock vs FLOPs-based savings | Sparse channel training · CIFAR-10 VGG-19 | speedup (x): 44.68x cost savings vs 7.41x time | Mixed |
Closest papers by meaning
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.89)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.89)
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.88)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.88)
- Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes (AAAI 2024 · similarity 0.88)
- Dynamic Sparse Training with Structured Sparsity (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.