Research map / Architectures and efficiency / Pruning and efficient tuning
Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sparse Cocktail, a sparse co-training framework that jointly trains unstructured, channel-wise, and N:M sparse subnetworks across multiple sparsity ratios from one training run, using unified mask generation, dense pivot co-training, and sparse network interpolation.
Problem
Existing sparse co-training methods only co-train one sparsity pattern per run (or a dense/sparse pair), so they cannot supply subnetworks across diverse sparsity patterns and ratios for varying hardware and resource constraints.
Approach
Alternately trains subnetworks of different sparsity patterns (unstructured, channel-wise, N:M) with gradually increasing sparsity ratios across iterative pruning with rewinding, generating all masks via a unified magnitude-based criterion (UMG), inserting dense pivot steps with dynamic distillation between sparse updates, and merging subnetworks via greedy interpolation.
Key results
- On ResNet-50 ImageNet, Sparse Cocktail reaches 76.32% average accuracy for dense networks and 73.23% for unstructured sparse subnetworks, comparable to AST (73.15%) and AC/DC (75.80%) for unstructured.
- Sparse Cocktail co-trains 24 subnetworks (1 dense, 10 unstructured, 10 channel-wise, 3 N:M) with 1x parameter count, versus 1.94x for AST and 10x for AC/DC.
- On MS COCO with ResNet50, Sparse Cocktail gets 31.0 BoxAP vs MutualNet 31.3, while on VGG-16 it gets 32.3 BoxAP vs MutualNet 30.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add weight rewinding after pruning (ablation) | Sparse Cocktail · CIFAR10 and ImageNet, all reported sparsity patterns | accuracy | Helped |
| Add network interpolation across IMP stages (ablation) | Sparse Cocktail · CIFAR10 and ImageNet, all reported sparsity patterns | accuracy | Helped |
| Use Dense Pivot Co-training instead of AST (ablation) | Sparse Cocktail · CIFAR10 and ImageNet, all reported sparsity patterns | accuracy | Helped |
| Use Unified Mask Generation instead of vanilla mask generation (ablation) | Sparse Cocktail · CIFAR10 and ImageNet, all reported sparsity patterns | accuracy | Helped |
| Co-train channel-wise subnetworks with Sparse Cocktail rather than MutualNet | ResNet-50 on ImageNet · ImageNet | average accuracy (%): 72.04 -> 72.22 | Helped |
Surprising findings
- Naively applying AST across mixed sparsity patterns degrades performance, which motivated the dense pivot step.
Limitations
- Only three sparsity patterns (unstructured, channel-wise, N:M) are covered; other patterns are left to future work.
- Training is multi-phase iterative pruning with rewinding, which is costlier than single-pass training and not designed for training efficiency.
- Some individual results trail baselines (e.g., channel-wise and N:M accuracy below AC/DC/AST on ImageNet; MutualNet higher BoxAP on ResNet50).
Experimental setup
Datasets: CIFAR-10 ImageNet MS COCO
Benchmarks: CIFAR-10 ImageNet MS COCO
Metrics: top-1 accuracy average accuracy over sparsity ratios BoxAP MaskAP
Baselines: MutualNet AC/DC AST
Builds on: Alternate Sparse Training (AST) AC/DC MutualNet Iterative Magnitude Pruning weight refilling
Closest papers by meaning
- Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together! (ICLR 2023 · similarity 0.91)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.88)
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.88)
- Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization (ICLR 2025 · similarity 0.87)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.87)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.