Research map / Large language models / Language, safety and interpretability
PROXSPARSE: REGULARIZED LEARNING OF SEMI-STRUCTURED SPARSITY MASKS FOR PRETRAINED LLMS
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProxSparse, a learned 2:4 semi-structured mask selection method that uses regularized proximal gradient descent with an efficient enumeration-based proximal solver, achieving better perplexity and zero-shot accuracy than Wanda and SparseGPT on seven LLMs.
Problem
Finding effective 2:4 semi-structured sparsity masks for pretrained LLMs without extensive retraining; existing heuristic local methods (Wanda, SparseGPT) do not leverage global end-to-end feedback from calibration data.
Approach
Relax the hard 2:4 sparsity and frozen-weight constraints into regularizers (a product-of-absolute-values 2:4 regularizer and a weighted distance-to-pretrained regularizer), solve with proximal gradient descent whose proximal operator is computed by an alternating-minimization enumeration solver (EnumALM), and then read off the mask from the learned weights without further weight updates.
Key results
- On Llama-2-7b, ProxSparse reaches 8.51 Wikitext PPL vs Wanda 11.42 and SparseGPT 10.30 at 2:4 sparsity.
- On Mistral-v0.1-7b, ProxSparse averages 52.7% zero-shot accuracy vs Wanda 44.1%.
- Sparse model yields 1.26x end-to-end inference speedup and 29.5%-37.3% peak memory reduction on A100 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ProxSparse instead of SparseGPT for 2:4 pruning | Mistral-v0.1-7b · Wikitext | perplexity: 9.43 -> 8.92 | Helped |
| Use ProxSparse instead of SparseGPT for 2:4 pruning | Qwen2.5-14B · Zero-shot tasks | AVG accuracy: 0.507 -> 0.525 | Helped |
| Use ProxSparse instead of SparseGPT for 2:4 pruning | Llama-2-7b · Wikitext | perplexity: 10.30 -> 8.51 | Helped |
| Relax both sparsity and frozen-weight constraints rather than using hard constraints (ablation) | ProxSparse · Wikitext, Mistral-v0.3-7b | perplexity: 13.6 -> 8.68 | Helped |
| Relax both sparsity and frozen-weight constraints rather than using hard constraints (ablation) | ProxSparse · Wikitext, OpenLlama-7b-v2 | perplexity: 35.28 -> 9.91 | Helped |
| Add sparsity regularization at a balanced strength (λ1=0.5) rather than λ1=inf (ablation) | ProxSparse · Wikitext, Mistral-v0.3-7b | perplexity: 11.33 -> 8.68 | Helped |
| Increase sparsity regularization from λ1=0.001 to λ1=0.5 (ablation) | ProxSparse · Wikitext, Mistral-v0.3-7b | perplexity: 14.19 -> 8.68 | Helped |
| Increase frozen-weight regularization from λ2=0 to λ2=inf (ablation) | ProxSparse · Wikitext, OpenLlama-7b-v2 | perplexity: 9.91 -> 34 | Hurt |
| Increase calibration samples from 100 to 400 (ablation) | ProxSparse on OpenLlama-7b-v2 · Wikitext | perplexity: 10.39 -> 9.91 | Helped |
| Increase calibration samples from 100 to 400 (ablation) | ProxSparse on Llama-2-7b · Wikitext | perplexity: 9.24 -> 8.51 | Helped |
| Use EnumALM instead of EnumIPM for the proximal solver | ProxSparse solver · 100 randomly generated problem instances | total runtime (sec): 561.70 -> 8.52 | Helped |
| Use EnumALM instead of EnumPGD for the proximal solver | ProxSparse solver · 100 randomly generated problem instances | total runtime (sec): 43.31 -> 8.52 | Helped |
Surprising findings
- Hard (projected) constraints during optimization perform worst; relaxing both sparsity and frozen-weight constraints gives the best perplexity.
- Weight-update-free ProxSparse outperforms SparseGPT, which does reconstruction updates, on most models.
Limitations
- Only 2:4 sparsity evaluated in main experiments; extension to other patterns (e.g., 4:8) left to appendix discussion and may increase search cost.
- Hyperparameters (lambda1, lambda2, learning rate) are tuned per model, with lambda1 requiring careful balance to avoid under- or over-regularization.
- Convergence guarantee to a critical point relies on assumptions (bounded iterates, smooth loss) and inner solver tolerance.
Experimental setup
Datasets: C4
Benchmarks: WikiText perplexity ARC-C ARC-E SIQA HellaSwag OBQA PIQA TruthfulQA
Metrics: perplexity accuracy speedup peak memory reduction
Baselines: magnitude pruning SparseGPT Wanda MaskLLM ADMMPrune OWL AlphaPrune
Builds on: Wanda SparseGPT proximal gradient descent EnumIPM
Closest papers by meaning
- MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models (NeurIPS 2024 · similarity 0.90)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.89)
- MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs (ICLR 2026 · similarity 0.89)
- Instruction-Following Pruning for Large Language Models (ICML 2025 · similarity 0.88)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.88)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.