Research map / Architectures and efficiency / Pruning and efficient tuning
Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DCTpS instead of standard sparse layers | ResNet50 · CIFAR100, 0.0001 density | Δ accuracy vs dense baseline: -65.7% (RigL (ERK)) -> -22.8% | Helped |
| Combine DCTpS with RigL | DCTpS · ResNet50 on CIFAR100, 0.01% density | accuracy: 44% -> ≈ 50% | Helped |
| Combine DCTpS with RigL | DCTpS · VGG19 on CIFAR100 | accuracy: 3% - 6% | Helped |
| Use DCTpS instead of static sparse initialization | MobileNetV2 · CIFAR10, 0.01% density | accuracy | Helped |
| Use DCTpS instead of FORCE or SynFlow | ResNet18 · Tiny ImageNet, densities ≤ 1% | validation accuracy: approximately 10% | Helped |
| Replace equal-per-layer support distribution with equal-per-filter support distribution (ablation) | DCTpS · ResNet50 and VGG19 | accuracy | Hurt |
| Fix α at 1 instead of training it (ablation) | DCTpS · Experiments across tested architectures and densities | accuracy: 2%-3% | Hurt |
| Use SGD with momentum instead of Adam (ablation) | DCTpS · ResNet50, VGG19, MobileNetV2, and FixupResNet110 | test accuracy | Mixed |
| Use equal-per-layer allocation instead of uniform random pruning | Random sparse initialization · ResNet50 and VGG19 experiments | accuracy | Helped |
Closest papers by meaning
- The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training (ICLR 2022 · similarity 0.92)
- Training Your Sparse Neural Network Better with Any Mask (ICML 2022 · similarity 0.92)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.89)
- Sparser, Better, Deeper, Stronger: Improving Static Sparse Training with Exact Orthogonal Initialization (ICML 2024 · similarity 0.89)
- Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities (ICML 2024 · similarity 0.89)
- Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.