Research map / Architectures and efficiency / Pruning and efficient tuning
SequentialAttention++ for Block Sparsification: Differentiable Pruning Meets Combinatorial Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unifies differentiable pruning (softmax/attention masks, powerpropagation) with combinatorial local search (ACDC-style IHT) via nonconvex group regularization, and proposes SequentialAttention++ for block-sparse pruning of ResNet50 on ImageNet and an MLP on Criteo.
Problem
Structured (block) neural network pruning requires both accurate importance scoring of weight blocks and effective search over sparse supports; these have been developed separately, with little theory explaining differentiable pruning methods.
Approach
Shows many differentiable pruning masks equal nonconvex group regularizers whose global minimizer is unique, group-sparse and matches group LASSO's 1-sparse solution; uses softmax attention over blocks inside an ACDC-style alternating dense/sparsification/sparse local search.
Key results
- On ImageNet ResNet50 with 64x64 blocks at 79% sparsity, SequentialAttention++ reaches 70.30 vs ACDC 67.36 (dense baseline 76.90).
- On ImageNet 16x16 blocks at 95% sparsity, SequentialAttention++ achieves 64.27 vs ACDC 61.42.
- On Criteo at 99% sparsity with block size 5, SequentialAttention++ attains validation loss 0.4803 vs ACDC 0.5050 and magnitude 0.4881.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SequentialAttention++ instead of ACDC | ResNet50 block sparsification · ImageNet, tested block sizes and sparsities | validation accuracy: 74.11 -> 74.14; 72.47 -> 72.90; 67.74 -> 69.56; 74.08 -> 74.40; 72.56 -> 73.50; 68.61 -> 69.92; 61.42 -> 64.27; 74.40 -> | Helped |
| Use SequentialAttention++ instead of the strongest listed baseline | Criteo block sparsification · Criteo, 90% sparsity, block sizes 5, 10, and 20 | validation loss: 0.4517 -> 0.4515; 0.4572 -> 0.4535; 0.4829 -> 0.4596 | Helped |
| Add the sparsification phase (ablation) | SequentialAttention++ on ResNet50 · ImageNet, block-sparse settings | validation accuracy: -0.0235; -0.04006; -0.00624; -0.01408; -0.01262; -0.00738; +0.00089; -0.00569; -0.00837; -0.00429; -0.00196; +0.00059; - | Helped |
| Set sparsification schedule exponent to c=4 rather than c=2 or c=8 (ablation) | SequentialAttention++ on ResNet50 · ImageNet, 90% sparsity, block sizes 8×8, 16×16, 32×32, and 64×64 | validation accuracy: 0.69403 -> 0.6956; 0.69613 -> 0.6992; 0.70614 -> 0.70817; 0.72264 -> 0.72756; 0.69202 -> 0.6956; 0.70036 -> 0.6992; 0.70 | Mixed |
Surprising findings
- Removing the sparsification phase lowers accuracy by about 0.9 points on average, and in some settings (e.g., 64x64 blocks at 58%) the change is slightly positive, showing mixed effects.
Limitations
- Theory only establishes guarantees for the global minimizer of nonconvex-regularized problems, not local minima or convergence of the practical algorithm.
- Theory assumes strictly convex/strongly convex differentiable losses; deep network training is nonconvex.
- Adds one trainable parameter per block, which is a notable overhead at small block sizes.
Experimental setup
Datasets: ImageNet Criteo
Benchmarks: ImageNet Criteo
Metrics: validation accuracy validation loss AUC
Baselines: Magnitude pruning Powerpropagation ACDC
Builds on: Sequential Attention ACDC Powerpropagation group LASSO iterative hard thresholding
Closest papers by meaning
- The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training (ICLR 2022 · similarity 0.90)
- Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities (ICML 2024 · similarity 0.89)
- SAFE: Finding Sparse and Flat Minima to Improve Pruning (ICML 2025 · similarity 0.88)
- Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization (ICLR 2025 · similarity 0.88)
- How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective (NeurIPS 2024 · similarity 0.87)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.