Research map / Architectures and efficiency / Pruning and efficient tuning
ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALPS, an ADMM-based operator-splitting framework for ℓ0-constrained layer-wise one-shot LLM pruning, followed by a GPU-vectorized preconditioned conjugate gradient refinement on the fixed support.
Problem
One-shot unstructured pruning of LLMs relies on heuristics (magnitude, Wanda, SparseGPT, DSnoT) that yield suboptimal layer-wise reconstruction, especially at high sparsity.
Approach
Formulate layer-wise pruning as an ℓ0-constrained least-squares problem and solve it jointly for support and weights using ADMM with an adaptive penalty (rho) schedule tied to support changes, then refine weights on the fixed support with a vectorized, GPU-parallel PCG solver.
Key results
- On LLaMA3-8B at 70% sparsity, ALPS gives a 29% reduction in WikiText2 test perplexity versus competing methods (abstract).
- On OPT-13B at 70% unstructured sparsity, ALPS reaches WikiText2 perplexity 16.71 vs 19.29 for SparseGPT and 63.42 for Wanda.
- On OPT-13B k_proj layer at 0.8 sparsity, ALPS relative reconstruction error is 5.32e-2 vs 6.95e-2 for SparseGPT (support quality table, 20-40% lower error).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace SparseGPT with ALPS pruning | OPT-30B at 70% sparsity · WikiText2 | perplexity: 13.61($\pm$0.22) -> 12.67($\pm$0.23) | Helped |
| replace SparseGPT with ALPS pruning | OPT-30B at 70% sparsity · PTB | perplexity: 18.94($\pm$0.25) -> 17.47($\pm$0.27) | Helped |
| replace SparseGPT with ALPS pruning | OPT-30B at 70% sparsity · C4 | perplexity: 13.96($\pm$0.35) -> 13.00($\pm$0.32) | Helped |
| replace SparseGPT with ALPS pruning | OPT-30B at 70% sparsity · LAMBADA | accuracy: 60.50($\pm$0.88) -> 63.52($\pm$1.38) | Helped |
| replace SparseGPT with ALPS pruning | OPT-30B at 70% sparsity · PIQA | accuracy: 73.74($\pm$0.27) -> 75.05($\pm$0.38) | Helped |
| replace SparseGPT with ALPS pruning | OPT-30B at 70% sparsity · ARC-Easy | accuracy: 63.26($\pm$0.44) -> 65.52($\pm$0.75) | Helped |
| replace SparseGPT with ALPS pruning | OPT-30B at 70% sparsity · ARC-Challenge | accuracy: 28.81($\pm$0.40) -> 30.55($\pm$0.76) | Helped |
| use ALPS instead of SparseGPT | OPT-30B with 2:4 sparsity · WikiText2 | perplexity: 10.90($\pm$0.07) -> 10.64($\pm$0.09) | Helped |
| use ALPS instead of SparseGPT | OPT-30B with 4:8 sparsity · ARC-Challenge | accuracy: 32.17($\pm$0.88) -> 32.01($\pm$0.49) | Mixed |
| add PCG weight refinement after magnitude pruning (ablation) | OPT-13B layer at sparsity 0.8 · MP-determined support | relative reconstruction error: 1.98e-1 -> 8.63e-2 | Helped |
| use PCG refinement rather than backsolve | OPT-13B layer at sparsity 0.8 · MP-determined support | runtime: 37.8 -> 0.77 | Helped |
| optimize the support end-to-end with ALPS rather than ADMM-Grad | OPT-13B layer at sparsity 0.7 · layer reconstruction | relative reconstruction error: 3.36e-2 -> 2.77e-2 | Helped |
Surprising findings
- ADMM with a fixed penalty rho either converges to poor solutions (large rho) or fails to converge (small rho), motivating the adaptive rho scheme.
- ALPS runtime is substantially longer than baselines (e.g., 14323 s vs 2980 s for SparseGPT on OPT-13B), a trade-off the authors accept.
Limitations
- ALPS runtime is several times higher than heuristic baselines such as SparseGPT and Wanda.
- Convergence theory is for a non-convex, non-continuous problem and the rho schedule is hand-tuned with empirically chosen constants.
- Performance on MMLU degrades significantly at high sparsity even with ALPS, so one-shot pruning alone is insufficient.
Experimental setup
Datasets: C4
Benchmarks: WikiText2 PTB C4 LAMBADA PIQA ARC-Easy ARC-Challenge MMLU
Metrics: perplexity accuracy relative reconstruction error
Baselines: Magnitude Pruning SparseGPT Wanda DSnoT ADMM-Grad
Builds on: ADMM SparseGPT Wanda DSnoT
Closest papers by meaning
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.92)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.92)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.92)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.91)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.91)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.