Research map / Architectures and efficiency / Pruning and efficient tuning
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, Christopher Re
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Pixelated Butterfly sparse training | MLP-Mixer and ViT · ImageNet, CIFAR-10/100 | training speedup: up to 2.3× | Improved |
| use Pixelated Butterfly | GPT-2 small · WikiText-103 | training speedup: 2.1× | Improved |
| use Pixelated Butterfly | GPT-2 medium · WikiText-103 | training speedup: up to 2.5× | Improved |
| use Pixelated Butterfly | Transformer · Long Range Arena | training speedup: 5.2× | Improved |
| replace existing hand-crafted sparsity patterns with butterfly sparsity (ablation) | Pixelated Butterfly · ImageNet | accuracy: up to 2% | Improved |
| align sparsity to hardware blocks (ablation) | Pixelated Butterfly · Sparse training | training speedup: up to 5× | Improved |
| allocate compute budget across all layers according to their compute share (ablation) | ViT-small · ViT-small | training speedup: 2× | Improved |
| allocate about one-quarter of the budget to low-rank and the remainder to flat block butterfly (ablation) | Pixelated Butterfly Mixer-S · CIFAR-10 | accuracy | Improved |
| use flat block butterfly instead of regular butterfly (ablation) | Pixelated Butterfly · GPU multiplication | multiplication speed: up to 3× | Improved |
| sparsify both attention and MLP rather than only one component (ablation) | ViT-small · ViT-small | end-to-end training speedup | Improved |
| use the smallest hardware-supported block size (ablation) | Pixelated Butterfly · Sparse matrix multiplication | training speed | Improved |
| use Pixelated Butterfly instead of dense training | Mixer-S/32 · CIFAR-10 | accuracy | Improved |
Closest papers by meaning
- Deformable Butterfly: A Highly Structured and Sparse Linear Transform (NeurIPS 2021 · similarity 0.89)
- Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization (ICLR 2025 · similarity 0.88)
- Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once (ICML 2024 · similarity 0.87)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.86)
- SequentialAttention++ for Block Sparsification: Differentiable Pruning Meets Combinatorial Optimization (NeurIPS 2024 · similarity 0.86)
- Navigating Extremes: Dynamic Sparsity in Large Output Spaces (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
