AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

AC/DC on a Budget -- Alternating Sparse Phases

Rahul Nittala, Advait Gadhikar, Tom Jacobs, Rebekka Burkholz

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Sparse training from scratch · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes why alternating dense and sparse training phases (AC/DC) help neural network sparsification, finding that reintroduced masked parameters get disproportionately large gradients that drive mask rewiring and sign flips, and builds on this to propose a cheaper variant.

Problem

Alternating dense and sparse training (AC/DC) yields state-of-the-art neural network sparsification but is computationally expensive because of the dense phases.

Approach

Empirical analysis shows that when masked parameters are reintroduced in a dense phase they receive disproportionately large gradients, which drive mask rewiring and parameter sign-flipping; this mechanism is used to design a cheaper alternating sparse-phase scheme.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
zero-initialize masked parameters in dense phases (ablation)AC/DC · CIFAR-100 (95% sparsity, ResNet18)top-1 accuracy (%): 73.77 -> 74.86Improved
zero-initialize masked parameters in dense phases (ablation)AC/DC · ImageNet (70% sparsity, DeiT-Small)top-1 accuracy (%): 75.89 -> 75.891No clear effect
zero-initialize rewired parameters (ablation)ARC · ImageNet (70% sparsity, DeiT-Small)top-1 accuracy (%): 76.50 -> 76.21No clear effect
zero-initialize rewired parameters (ablation)ARC · ImageNet (95% sparsity, ResNet50)top-1 accuracy (%): 72.083 -> 73.087Improved
zero-initialize rewired parameters (ablation)ARC · GPT-2 on OpenWebText (80% sparsity)perplexity: 21.3395 -> 21.2521Improved
shorten dense phases from 10 epochs to as few as 3 (ablation)AC/DC · CIFAR-100 (90% sparsity, ResNet18)top-1 accuracy (%)No clear effect
increase density of the added high-gradient set G_t from 0.1 to 0.3 (with zero init) (ablation)ARC · CIFAR-100 (90% sparsity, ResNet18)top-1 accuracy (%): 74.74 -> 75.21Improved
replace dense phases with sparse rewiring phase (ARC 0.1)AC/DC · CIFAR-100 (95% sparsity, ResNet18)top-1 accuracy (%): 73.77 -> 73.91No clear effect
replace dense phases with sparse rewiring phase (ARC 0.2)AC/DC · ImageNet (70% sparsity, DeiT-Small)top-1 accuracy (%): 75.89 -> 76.50Improved
replace dense phases with sparse rewiring (ARC-Zero)AC/DC · ImageNet (95% sparsity, ResNet50)training FLOPs (x dense): 0.53x -> 0.2431xImproved
replace dense phases with sparse rewiring (ARC-Zero)AC/DC · GPT-2 on OpenWebText (80% sparsity)perplexity: 23.0327 -> 21.2521Improved
replace dense phases with sparse rewiring (ARC-Zero)AC/DC · ImageNet (95% sparsity, ResNet50)top-1 accuracy (%): 73.14 -> 73.087No clear effect

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Surprising findings

Experimental setup

Baselines: AC/DC

Builds on: AC/DC

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy