Research map / Architectures and efficiency / Pruning and efficient tuning
AC/DC on a Budget -- Alternating Sparse Phases
Rahul Nittala, Advait Gadhikar, Tom Jacobs, Rebekka Burkholz
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why alternating dense and sparse training phases (AC/DC) help neural network sparsification, finding that reintroduced masked parameters get disproportionately large gradients that drive mask rewiring and sign flips, and builds on this to propose a cheaper variant.
Problem
Alternating dense and sparse training (AC/DC) yields state-of-the-art neural network sparsification but is computationally expensive because of the dense phases.
Approach
Empirical analysis shows that when masked parameters are reintroduced in a dense phase they receive disproportionately large gradients, which drive mask rewiring and parameter sign-flipping; this mechanism is used to design a cheaper alternating sparse-phase scheme.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| zero-initialize masked parameters in dense phases (ablation) | AC/DC · CIFAR-100 (95% sparsity, ResNet18) | top-1 accuracy (%): 73.77 -> 74.86 | Improved |
| zero-initialize masked parameters in dense phases (ablation) | AC/DC · ImageNet (70% sparsity, DeiT-Small) | top-1 accuracy (%): 75.89 -> 75.891 | No clear effect |
| zero-initialize rewired parameters (ablation) | ARC · ImageNet (70% sparsity, DeiT-Small) | top-1 accuracy (%): 76.50 -> 76.21 | No clear effect |
| zero-initialize rewired parameters (ablation) | ARC · ImageNet (95% sparsity, ResNet50) | top-1 accuracy (%): 72.083 -> 73.087 | Improved |
| zero-initialize rewired parameters (ablation) | ARC · GPT-2 on OpenWebText (80% sparsity) | perplexity: 21.3395 -> 21.2521 | Improved |
| shorten dense phases from 10 epochs to as few as 3 (ablation) | AC/DC · CIFAR-100 (90% sparsity, ResNet18) | top-1 accuracy (%) | No clear effect |
| increase density of the added high-gradient set G_t from 0.1 to 0.3 (with zero init) (ablation) | ARC · CIFAR-100 (90% sparsity, ResNet18) | top-1 accuracy (%): 74.74 -> 75.21 | Improved |
| replace dense phases with sparse rewiring phase (ARC 0.1) | AC/DC · CIFAR-100 (95% sparsity, ResNet18) | top-1 accuracy (%): 73.77 -> 73.91 | No clear effect |
| replace dense phases with sparse rewiring phase (ARC 0.2) | AC/DC · ImageNet (70% sparsity, DeiT-Small) | top-1 accuracy (%): 75.89 -> 76.50 | Improved |
| replace dense phases with sparse rewiring (ARC-Zero) | AC/DC · ImageNet (95% sparsity, ResNet50) | training FLOPs (x dense): 0.53x -> 0.2431x | Improved |
| replace dense phases with sparse rewiring (ARC-Zero) | AC/DC · GPT-2 on OpenWebText (80% sparsity) | perplexity: 23.0327 -> 21.2521 | Improved |
| replace dense phases with sparse rewiring (ARC-Zero) | AC/DC · ImageNet (95% sparsity, ResNet50) | top-1 accuracy (%): 73.14 -> 73.087 | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.
- AC/DC — AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks (NeurIPS 2021) · better on 3 of 4 shared results
- Top-KAST — Top-KAST: Top-K Always Sparse Training (NeurIPS 2020) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Surprising findings
- Previously masked parameters receive disproportionately large gradients upon reintroduction in dense phases, driving mask rewiring and sign flipping.
Experimental setup
Baselines: AC/DC
Builds on: AC/DC
Closest papers by meaning
- Exploring the Benefit of Activation Sparsity in Pre-training (ICML 2024 · similarity 0.86)
- DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.86)
- Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency (ICML 2024 · similarity 0.85)
- BAME: Block-Aware Mask Evolution for Efficient N:M Sparse Training (ICML 2025 · similarity 0.85)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.85)
- Sparser, Better, Deeper, Stronger: Improving Static Sparse Training with Exact Orthogonal Initialization (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
