Research map / Architectures and efficiency / Pruning and efficient tuning
Training Your Sparse Neural Network Better with Any Mask
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ToST during LTH sparse-mask training | LTH masks, ResNet-32 and ResNet-50 · CIFAR-10 and CIFAR-100, 90%-98% sparsity | accuracy | Helped |
| Apply ToST to randomly pruned masks | randomly pruned ResNet-32 masks · CIFAR-100, 98% sparsity | accuracy: 19.83±3.21 -> 33.98±6.64 | Helped |
| Apply ToST during SNIP mask training | SNIP masks · TinyImageNet, ResNet-50, 95% sparsity | accuracy: 51.19±0.47 -> 53.21±0.08 | Helped |
| Add ghost skip connections (ablation) | LTH masks, ResNet-18 · CIFAR-100, 95% sparsity | accuracy: +2.15% | Helped |
| Add ghost soft neurons (ablation) | LTH masks, ResNet-18 · CIFAR-100, 95% sparsity | accuracy: +1.28% | Helped |
| Add layer-wise re-scaled initialization (ablation) | LTH masks, ResNet-18 · CIFAR-100, 75% sparsity | accuracy | Helped |
| Add label smoothing (ablation) | LTH masks, ResNet-18 · CIFAR-100, 80% sparsity | accuracy | Helped |
| Combine the ToST tweaks (ablation) | LTH masks, ResNet-18 · CIFAR-100, 75%-95% sparsity | accuracy: 1.08% - 2.51% | Helped |
| Keep ghost skip connections through training instead of removing them on schedule (ablation) | LTH masks · CIFAR-100, 95% sparsity | accuracy | Helped |
| Keep Swish through training instead of removing it on schedule (ablation) | LTH masks · CIFAR-100, 95% sparsity | accuracy | Helped |
| Remove GSk after the first learning-rate decay rather than before it (ablation) | LTH masks · CIFAR-100, 95% sparsity | accuracy: -0.917% | Hurt |
| Remove GSw after the first learning-rate decay rather than before it (ablation) | LTH masks · CIFAR-100, 95% sparsity | accuracy: -0.429% | Hurt |
Closest papers by meaning
- The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training (ICLR 2022 · similarity 0.94)
- Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers (ICLR 2020 · similarity 0.92)
- Why Random Pruning Is All We Need to Start Sparse (ICML 2023 · similarity 0.92)
- Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once (ICML 2024 · similarity 0.90)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.89)
- SAFE: Finding Sparse and Flat Minima to Improve Pruning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.