Research map / Architectures and efficiency / Pruning and efficient tuning
Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SWAMP, an IMP variant that trains multiple sparse particles from the same matching ticket with different SGD noise, uses SWA within each, and averages them into one mask per cycle, outperforming IMP and other pruning baselines on image classification and some language tasks.
Problem
Iterative magnitude pruning (IMP) with rewinding is state-of-the-art for finding sparse masks, but its solutions are unstable and it is weak at extreme sparsity; the paper asks whether averaging multiple linearly connected IMP particles can yield better sparse solutions.
Approach
Train N particles from the same matching ticket with different SGD noise, use stochastic weight averaging along each trajectory, average all particles' weights, and prune by magnitude of the averaged weights each cycle, so the single output mask and final model benefit from a flat, well-connected low-loss region at no extra inference cost.
Key results
- On CIFAR-10 WRN-28-2 at 95% sparsity, SWAMP reaches 94.23% accuracy versus 93.58% for IMP (3 seeds).
- On CIFAR-100 WRN-32-4 at 95% sparsity, SWAMP achieves 76.48% versus 74.60% for IMP.
- On RTE (RoBERTa) at 61.3% sparsity, SWAMP improves over IMP by 5.17 points (62.63 vs 57.46).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace IMP with SWAMP | WRN-28-2 · CIFAR-10, 50%–95% sparsity | accuracy: 93.97 ± 0.16 -> 94.74 ± 0.04; 94.02 ± 0.23 -> 94.88 ± 0.09; 93.90 ± 0.15 -> 94.73 ± 0.10; 93.58 ± 0.09 -> 94.23 ± 0.11 | Helped |
| replace IMP with SWAMP | WRN-32-4 · CIFAR-100, 50%–95% sparsity | accuracy: 75.40 ± 0.23 -> 77.29 ± 0.53; 75.72 ± 0.41 -> 77.35 ± 0.39; 75.24 ± 0.25 -> 77.14 ± 0.33; 74.60 ± 0.37 -> 76.48 ± 0.73 | Helped |
| replace IMP with SWAMP | ResNet-50 · ImageNet IN-Val, IN-V2, IN-R and IN-S at 88.9% sparsity | accuracy: 71.66±0.09 -> 71.81±0.19; 59.00±0.32 -> 59.44±0.07; 30.92±0.10 -> 32.21±0.73; 19.32±0.46 -> 20.08±0.42 | Helped |
| add SWA trajectory averaging (ablation) | SWAMP on WRN-28-2 · CIFAR-10, 50% and 95% sparsity | accuracy: 94.43 ± 0.10 -> 94.74 ± 0.04; 93.73 ± 0.32 -> 94.23 ± 0.11 | Helped |
| add multi-particle averaging (ablation) | SWAMP on WRN-28-2 · CIFAR-10, 50% and 95% sparsity | accuracy: 94.62 ± 0.06 -> 94.74 ± 0.04; 93.97 ± 0.10 -> 94.23 ± 0.11 | Helped |
| increase particle count from 1 to 8 (ablation) | SWAMP on WRN-28-2 · CIFAR-10, 50%, 75%, 90% and 95% sparsity | accuracy: 94.62 ± 0.06 -> 94.80 ± 0.04; 94.67 ± 0.06 -> 94.90 ± 0.09; 94.35 ± 0.06 -> 94.74 ± 0.10; 93.97 ± 0.10 -> 94.21 ± 0.24 | Mixed |
| use a SWAMP-generated mask instead of an IMP-generated mask (ablation) | sparse training on CIFAR-10/100 · fixed sparsity | accuracy | Mixed |
| use SWAMP training instead of SGD training (ablation) | fixed IMP or SWAMP masks · CIFAR-10/100 | accuracy | Helped |
| increase pruning ratio (ablation) | SWAMP · CIFAR-10 and CIFAR-100 | accuracy: 93.58±0.09 -> 94.68±0.04; 74.60±0.37 -> 76.50±0.40 | Helped |
| add SWAMP particle averaging to RigL (ablation) | WRN28x2 and VGG-13 · CIFAR-10, 75%–95% sparsity | accuracy: 93.09±0.13 -> 93.41±0.04; 91.81±0.18 -> 92.78±0.10; 90.73±0.16 -> 92.40±0.46; 93.18±0.04 -> 93.53±0.08; 92.46±0.16 -> 93 | Helped |
| use cost-efficient SWAMP+ instead of SWAMP (ablation) | WRN-28-2 · CIFAR-10, 95% sparsity | GFLOPs: 4.75 -> 1.39 | Helped |
Surprising findings
- Weight averaging of particles does not outperform individual particles at early IMP cycles (low sparsity, e.g., 20%), only at higher sparsity.
Limitations
- Training cost scales linearly with the number of particles, roughly 4x IMP's FLOPs for CIFAR.
- Weight averaging fails to help at low sparsity in early IMP cycles due to non-convex landscape.
- No theoretical analysis of why the convex hull of particles forms a low-loss subspace.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet MRPC RTE
Benchmarks: CIFAR-10 CIFAR-100 ImageNet ImageNet-V2 ImageNet-Rendition ImageNet-Sketch MRPC RTE
Metrics: classification accuracy negative log-likelihood Hessian trace F1
Baselines: IMP IMP + SAM Lottery Pools SNIP GraSP SynFlow RigL DST
Builds on: Iterative magnitude pruning with rewinding Stochastic Weight Averaging Linear mode connectivity of IMP solutions Lottery Pools
Closest papers by meaning
- Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging (ICLR 2024 · similarity 0.89)
- The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training (ICLR 2022 · similarity 0.86)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.86)
- Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities (ICML 2024 · similarity 0.86)
- SequentialAttention++ for Block Sparsification: Differentiable Pruning Meets Combinatorial Optimization (NeurIPS 2024 · similarity 0.86)
- SAFE: Finding Sparse and Flat Minima to Improve Pruning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.