Research map / Architectures and efficiency / Pruning and efficient tuning
Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sparse Model Soups (SMS), which averages IMP retraining runs that share a fixed pruning mask and restarts each prune-retrain phase from the averaged model, improving generalization, OOD robustness and pruning-during-training methods.
Problem
Averaging independently trained sparse models (model soups) reduces sparsity because different models have different sparse connectivities, and re-pruning the average degrades performance.
Approach
Retrain multiple copies of one pruned model with varied hyperparameters (seed, batch order, weight decay, length) so they share the mask, average them after each prune-retrain phase, and start the next phase from the average, so sparsity is preserved and generalization improves.
Key results
- On CIFAR-100 (WRN-20, 98% sparsity, 3 phases), SMS with uniform soup of 10 models reaches 73.05% vs 70.74% for IMP.
- On ImageNet (ResNet-50, 90% sparsity), SMS uniform with m=10 reaches 74.96% vs 73.59% for IMP.
- BIMP+SMS on ImageNet at 90% sparsity reaches 74.05% vs 73.53% for BIMP, and DPF+SMS reaches 74.31% (best) at 90% sparsity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Average retrained pruned models with uniform weights at each prune-retrain phase | IMP via SMS · CIFAR-100, 98% target sparsity, Phase 3, m=10 | accuracy: 72.21 -> 73.05 | Helped |
| Average retrained pruned models with uniform weights at each prune-retrain phase | IMP via SMS · ImageNet, 90% target sparsity, Phase 3, m=10 | accuracy: 74.25 -> 74.96 | Helped |
| Add SMS model averaging within BIMP phases | BIMP · ImageNet, 70% sparsity | accuracy: 75.62 -> 76.20 | Helped |
| Add SMS model averaging within BIMP phases | BIMP · ImageNet, 80% sparsity | accuracy: 75.08 -> 75.76 | Helped |
| Add SMS model averaging within BIMP phases | BIMP · ImageNet, 90% sparsity | accuracy: 73.53 -> 74.05 | Helped |
| Add SMS model averaging between pruning steps | GMP · ImageNet, 70% sparsity | accuracy: 74.55 -> 75.10 | Helped |
| Add SMS model averaging between pruning steps | GMP · ImageNet, 80% sparsity | accuracy: 73.92 -> 74.48 | Helped |
| Add SMS model averaging between pruning steps | GMP · ImageNet, 90% sparsity | accuracy: 72.81 -> 73.12 | Helped |
| Add SMS model averaging between pruning steps | DPF · ImageNet, 70% sparsity | accuracy: 75.74 -> 76.26 | Helped |
| Add SMS model averaging between pruning steps | DPF · ImageNet, 80% sparsity | accuracy: 75.27 -> 75.85 | Helped |
| Add SMS model averaging between pruning steps | DPF · ImageNet, 90% sparsity | accuracy: 73.88 -> 74.31 | Helped |
| Use uniform rather than greedy soup merging (ablation) | SMS · CIFAR-100, 98% target sparsity, Phase 3, m=10 | accuracy: 72.44 -> 73.05 | Helped |
Surprising findings
- Starting a retraining phase from a model soup yields better individual models than starting from a single model, even when the soup candidates are averaged only from the previous phase.
- Uniform soup collapses at high sparsity on CIFAR-100 (averaging degrades), while greedy soup performs at least as well as the best individual model.
Limitations
- Uniform soup performance collapses at high sparsity where retrained models diverge in basin.
- Compute scales by factor m versus regular IMP; SMS is on par with IMP_mx in total compute.
- Weight alignment/permutation only partially mitigates sparsity reduction in later phases, so SMS relies on consistent masks.
Experimental setup
Datasets: ImageNet-1K CIFAR-10 CIFAR-100 Celeb-A Cityscapes WMT16 DE-EN CIFAR-100-C ImageNet-C
Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 Cityscapes WMT16 DE-EN CIFAR-100-C ImageNet-C
Metrics: top-1 accuracy mean IoU BLEU theoretical speedup OOD accuracy
Baselines: IMP IMP_mx (m-times longer retraining) IMP-RePrune GMP DPF BIMP GSM DNW LC DST
Builds on: Model soups Iterative Magnitude Pruning Linear learning rate restarting (LLR/ALLR) GMP DPF BIMP
Closest papers by meaning
- Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities (ICML 2024 · similarity 0.90)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.89)
- Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruning (ICLR 2024 · similarity 0.89)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.88)
- DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models (ICLR 2025 · similarity 0.88)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.