Research map / Architectures and efficiency / Pruning and efficient tuning
Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sparse-IFT, a family of FLOP-matched (iso-FLOP) sparse transformations that replace dense layers to improve accuracy over dense models, paired with dynamic sparse training and a Ramanujan-graph spectral analysis.
Problem
Sparse weight training reduces training FLOPs but usually lags dense accuracy, requiring longer schedules; the paper asks whether sparsity can instead improve a dense model's accuracy at equal FLOPs.
Approach
Replace each dense layer with an iso-FLOP sparse transformation (Sparse Wide, Sparse Parallel, Sparse Factorized, Sparse Doped) whose width, branch count, or rank is scaled with sparsity, enlarging the sparse mask-weight search space so dynamic sparse training (RigL) can find better subnetworks without changing training hyperparameters.
Key results
- Sparse Wide ResNet-18 at 90% sparsity reaches 74.4% top-1 on ImageNet vs 70.9% dense (+3.5%), matching dense ResNet-34 (74.2%) with 2x fewer training FLOPs.
- Sparse Wide IFT on RetinaNet/COCO improves AP from 29.3 to 34.5 at 90% sparsity; Cityscapes mIoU rises from 76.7 to 79.1.
- Sparse Wide IFT GPT-3 Small at 75% sparsity improves Open LLM leaderboard average from 33.8 to 34.7 (+0.9%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace dense ResNet-18 layers with Sparse Wide IFT at 90% sparsity | ResNet-18 · ImageNet | top-1 accuracy: 70.9 ± 0.1 -> 74.4 ± 0.2 | Helped |
| Replace dense ResNet-34 layers with Sparse Wide IFT at 90% sparsity | ResNet-34 · ImageNet | top-1 accuracy: 74.2 ± 0.1 -> 76.8 ± 0.3 | Helped |
| Replace dense BotNet-50 layers with Sparse Wide IFT at 90% sparsity | BotNet-50 · ImageNet | top-1 accuracy: 77.5 ± 0.1 -> 78.6 ± 0.3 | Helped |
| Use Sparse Wide IFT ResNet-18 backbone at 90% sparsity | ResNet-18 RetinaNet backbone · MS COCO 2017 minival | AP: 29.3 ± 0.1 -> 34.5 ± 0.2 | Helped |
| Use Sparse Wide IFT ResNet-18 backbone at 90% sparsity | ResNet-18 DeepLabV3+ backbone · Cityscapes val | mIoU: 76.7 ± 0.2 -> 79.1 ± 0.2 | Helped |
| Replace dense GPT-3 Small layers with Sparse Wide IFT at 75% sparsity | GPT-3 Small · Open LLM leaderboard (five-task average) | accuracy: 33.8 ± 0.1 -> 34.7 ± 0.2 | Helped |
| Use RigL instead of static sparse training for Sparse Wide IFT (ablation) | ResNet-18 Sparse Wide · CIFAR-100 | accuracy: 78.2 ± 0.3 -> 80.1 ± 0.2 | Helped |
| Add nonlinear activations to Sparse Factorized IFT (ablation) | ResNet-18 Sparse Factorized (90%) · CIFAR-100 | accuracy: 76.5 ± 0.4 -> 78.9 ± 0.5 | Helped |
| Add nonlinear activations to Sparse Parallel IFT (ablation) | ResNet-18 Sparse Parallel (90%) · CIFAR-100 | accuracy: 77.6 ± 0.1 -> 78.2 ± 0.2 | Helped |
| Add nonlinear activations to Sparse Doped IFT (ablation) | ResNet-18 Sparse Doped (90%) · CIFAR-100 | accuracy: 76.5 ± 0.2 -> 76.9 ± 0.2 | Helped |
| Use N:M block sparsity instead of unstructured sparsity (ablation) | ResNet-18 Sparse Wide · CIFAR-100 | accuracy: 80.1 -> 78.4 | Hurt |
| Extend Sparse Wide RigL training from 200 epochs to 200 · 1/(1-s) epochs at 90% sparsity (ablation) | ResNet-18 Sparse Wide · CIFAR-100 | accuracy: 80.1 ± 0.2 -> 76.4 ± 0.2 | Hurt |
Surprising findings
- Sparse-IFT with dynamic sparse training outperforms dense baselines at equal training FLOPs, not just static sparse baselines.
- Non-linear activations are essential: removing them drops Sparse Factorized at 90% sparsity to 76.5%, below the 77% dense baseline.
Limitations
- Wall-clock gains depend on hardware with unstructured sparsity support; on standard GPUs/CPUs overhead is large (19.5x inference latency at 90% sparsity without sparse kernels).
- Sparse Doped fails to gain accuracy at higher sparsity because its unstructured search space shrinks.
- Ramanujan-graph analysis is only performed on ResNet-18 at 50% sparsity on CIFAR-100, so its generality is limited.
Experimental setup
Datasets: CIFAR-100 ImageNet MS COCO 2017 Cityscapes The Pile
Benchmarks: CIFAR-100 ImageNet MS COCO Cityscapes Open LLM Leaderboard (ARC, HellaSwag, MMLU, TruthfulQA, Winogrande)
Metrics: top-1 accuracy AP mIoU mAcc average accuracy latency training throughput
Baselines: Static sparse SNIP GraSP FORCE SET RigL GraNet Dense ResNet-18 Dense ResNet-34
Builds on: RigL Lottery Ticket Hypothesis SET GraNet Chinchilla
Closest papers by meaning
- Dynamic Sparse Training with Structured Sparsity (ICLR 2024 · similarity 0.88)
- Advancing Dynamic Sparse Training by Exploring Optimization Opportunities (ICML 2024 · similarity 0.88)
- Navigating Extremes: Dynamic Sparsity in Large Output Spaces (NeurIPS 2024 · similarity 0.87)
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.87)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.86)
- Sparser, Better, Deeper, Stronger: Improving Static Sparse Training with Exact Orthogonal Initialization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.