Research map / Architectures and efficiency / RNNs and normalization
Sparse Weight Activation Training
Md Aamir Raihan, Tor Aamodt
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| SWAT-U instead of other sparse training methods at 80% weight sparsity | ResNet-50 · ImageNet val | top-1 accuracy (%): 75.2 vs 74.9 (SNFS), 74.6 (RigL) | Improved |
| sparsify weights and activations in both forward and backward pass (SWAT-U at 80%) | ResNet-50 · ImageNet val | top-1 accuracy change: -1.6 accuracy change at 76.1% FLOPs reduction | Hurt |
| use ERK layerwise sparsity instead of uniform (ablation) | SWAT ResNet-50 · ImageNet val 80% sparsity | top-1 accuracy (%): 75.2 -> 76.0 | Improved |
| structured SWAT channel pruning at 70% sparsity | ResNet-18, DenseNet-121 · CIFAR-10 | top-1 accuracy: +0.06 and +0.30 accuracy change | Improved |
| structured SWAT channel pruning at 70% sparsity | ResNet-50 · ImageNet val | top-1 accuracy (%): 76.80 -> 75.67 | Hurt |
| mask gradient updates for non-active weights (ablation) | SWAT ResNet-18 · CIFAR-100 | validation accuracy | Hurt |
| freeze sparse topology early in training (ablation) | SWAT ResNet-18 · CIFAR-100 | validation accuracy | Hurt |
| Top-K selection instead of random selection of components (ablation) | SAW ResNet-18 · CIFAR-100 | accuracy | Improved |
| constrain Top-K per channel (TOPK-HW) instead of whole tensor (ablation) | SAW ResNet-18 · CIFAR-100 | accuracy | Hurt |
| compute Top-K threshold periodically (every 100 iterations) (ablation) | SWAT ResNet-18 · CIFAR-100, 90% target sparsity | sparsity stability | No clear effect |
| sparsify output activation gradients (meProp) instead of weights and activations (SAW) (ablation) | ResNet-18 · ImageNet | convergence/accuracy | Hurt |
Closest papers by meaning
- Inducing and Exploiting Activation Sparsity for Fast Inference on Deep Neural Networks (ICML 2020 · similarity 0.90)
- Toward the Connection Between Activation Sparsity and Flat Minima (TPAMI 2026 · similarity 0.87)
- DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.86)
- Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once (ICML 2024 · similarity 0.85)
- Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency (ICML 2024 · similarity 0.85)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
