Research map / Architectures and efficiency / Pruning and efficient tuning
Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Double Sparse Factorization (DSF), which replaces each weight matrix with a product of two sparse matrices found via ADMM-based alternating minimization, improving one-shot layer-wise pruning of LLMs and CNNs.
Problem
Post-training neural network compression: sparsifying weight matrices more effectively than single-matrix magnitude or layer-wise pruning, with minimal accuracy loss at high sparsity for LLMs and CNNs.
Approach
Factorize each dense weight matrix into a product of two sparse matrices (AB with total nonzeros fixed), solved heuristically by alternating minimization where each L0-constrained subproblem is handled by ADMM with cubic annealing of the penalty, warm starting, and a layer-wise finalization step using calibration inputs.
Key results
- At 50% density, DSF reaches Wikitext-2 perplexity 4.87 on Llama2-13B, beating dense Llama2-7B (5.12) and ADMM (5.52).
- On ResNet-50 ImageNet at 2x FLOP reduction, DSF gets 75.78% vs OBC 75.65%; at 4x, 74.95% vs 74.05%.
- On CIFAR-10 ResNet-20-32 at 10% density without fine-tuning, DSF reaches 84.1% vs 51.1% for magnitude pruning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ordinary layer-wise pruning with double sparse factorization | LLaMA models at 50% density · Wikitext-2 | perplexity: 7.06 -> 6.12; 6.33 -> 5.58; 5.52 -> 4.87; 3.95 -> 3.44 | Helped |
| replace ordinary layer-wise pruning with double sparse factorization | LLaMA models at 30% density · Wikitext-2 | perplexity: 18.66 -> 8.33; 17.51 -> 8.01; 13.82 -> 6.43; 7.80 -> 4.56 | Helped |
| replace ADMM pruning with double sparse factorization | LLaMA-2-7B at 50% density · zero-shot task mean | accuracy: 56.64 -> 57.78 | Helped |
| replace ADMM pruning with double sparse factorization | LLaMA-2-13B at 50% density · zero-shot task mean | accuracy: 61.18 -> 62.54 | Helped |
| replace OBC layer-wise pruning with double sparse factorization | ResNet-50 · ImageNet, 2x FLOP reduction | test accuracy: 75.65 -> 75.78 | Helped |
| replace OBC layer-wise pruning with double sparse factorization | ResNet-50 · ImageNet, 3x FLOP reduction | test accuracy: 75.01 -> 75.56 | Helped |
| replace OBC layer-wise pruning with double sparse factorization | ResNet-50 · ImageNet, 4x FLOP reduction | test accuracy: 74.05 -> 74.95 | Helped |
| add the finalization step (ablation) | DSF on LLaMA-2-7B at 30% density · Wikitext-2 | perplexity: 9.13 -> 8.01 | Helped |
| fix one sparse mask to a shared random mask (ablation) | DSF on LLaMA-2-7B at 50% density · Wikitext-2 | perplexity: 5.58 -> 6.05 | Hurt |
| remove one sparse factor mask and use a shared random mask (ablation) | DSF · Wikitext-2 | perplexity | Hurt |
| use a cubic annealing schedule instead of setting the first rho to 1 (ablation) | DSF matrix projection · LLaMA-7B and ResNet-50 weight matrices | reconstruction error | Helped |
| initialize the first factor with the identity matrix (ablation) | DSF matrix projection · weight matrices | reconstruction error | Helped |
Surprising findings
- A 50%-pruned Llama2-13B with DSF has lower perplexity than the dense Llama2-7B, the first reported case of a uniformly layer-wise pruned model beating a smaller dense model per the authors.
- Two sparse multiplications are only about 10-20% slower on CPU than one sparse multiplication with the same nonzero count.
Limitations
- Layer-wise sparsity allocations must be set manually rather than determined globally as in magnitude pruning.
- Storage overhead from a second sparsity mask and extra inference latency of about 10-20%.
- Not clear how to integrate DSF with gradual pruning interleaved with full-network fine-tuning.
Experimental setup
Datasets: C4 Wikitext-2 CIFAR-10 ImageNet RedPajama BoolQ RTE HellaSwag WinoGrande ARC-e ARC-c OpenBookQA
Benchmarks: Wikitext-2 BoolQ RTE HellaSwag WinoGrande ARC-e ARC-c OpenBookQA CIFAR-10 ImageNet
Metrics: perplexity zero-shot accuracy test accuracy reconstruction error runtime
Baselines: Wanda SparseGPT ADMM pruning Optimal Brain Compression Magnitude pruning Cyclical pruning palm4msa Monarch decomposition SVD
Builds on: ADMM pruning Wanda Optimal Brain Compression Monarch matrices SparseGPT
Closest papers by meaning
- Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries (ICLR 2025 · similarity 0.89)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.88)
- SequentialAttention++ for Block Sparsification: Differentiable Pruning Meets Combinatorial Optimization (NeurIPS 2024 · similarity 0.88)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.88)
- Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once (ICML 2024 · similarity 0.87)
- Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.