Research map / Large language models / Language, safety and interpretability
AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AlphaPruning, which uses the heavy-tailed shape metric PL_Alpha_Hill from HT-SR theory to allocate layer-wise sparsity ratios for LLM pruning, improving Magnitude, Wanda, and SparseGPT at high sparsity.
Problem
Uniform layer-wise sparsity in LLM pruning limits achievable sparsity, and existing non-uniform allocation (e.g., OWL) relies on heuristics such as outlier counts that break down at high sparsity.
Approach
Allocate less sparsity to layers whose weight-matrix ESDs are more heavy-tailed (lower power-law exponent estimated by the Hill estimator), since these layers are better trained; shape metrics outperform scale metrics for this allocation.
Key results
- On LLaMA-7B at 70% sparsity with Wanda, AlphaPruning reduces WikiText perplexity from 85.77 (uniform) to 23.86, and at 80% sparsity it reduces perplexity by 304.31 versus OWL.
- AlphaPruning with DeepSparse on Llama2-7B-Chat reaches 3.06x end-to-end decode speedup on CPU at 80% global sparsity.
- Enabling AlphaPruning with LoRA fine-tuning on SparseGPT-pruned LLaMA-7B at 80% sparsity lowers perplexity from 20.11 (uniform) to 18.53.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace uniform layerwise sparsity with AlphaPruning for Wanda | Wanda pruning on LLaMA-7B at 70% sparsity · WikiText validation | perplexity: 85.77 -> 23.86 | Helped |
| Replace uniform layerwise sparsity with AlphaPruning for SparseGPT | SparseGPT pruning on LLaMA-7B at 70% sparsity · WikiText validation | perplexity: 26.30 -> 18.54 | Helped |
| Replace uniform layerwise sparsity with AlphaPruning for Magnitude pruning | Magnitude pruning on LLaMA-7B at 70% sparsity · WikiText validation | perplexity: 48419.13 -> 231.01 | Helped |
| Replace uniform layerwise sparsity with AlphaPruning for Wanda | Wanda pruning on LLaMA-7B at 70% sparsity · 7 zero-shot tasks | mean accuracy (%): 36.73 -> 44.42 | Helped |
| Replace uniform layerwise sparsity with AlphaPruning for SparseGPT | SparseGPT pruning on LLaMA-7B at 70% sparsity · 7 zero-shot tasks | mean accuracy (%): 41.52 -> 45.48 | Helped |
| Use PL_Alpha_Hill instead of a scale-based metric for layer allocation | Wanda pruning on LLaMA-7B at 70% sparsity · WikiText validation | perplexity: 59.82 -> 23.86 | Helped |
| Allocate sparsity per block instead of per matrix (ablation) | AlphaPruning with Wanda on LLaMA-7B at 70% sparsity · WikiText validation | perplexity: 58.71 -> 23.86 | Helped |
| Allocate sparsity per block instead of per matrix (ablation) | AlphaPruning with SparseGPT on LLaMA-7B at 70% sparsity · WikiText validation | perplexity: 25.10 -> 18.54 | Helped |
| Add per-matrix refinement to per-block allocation (ablation) | AlphaPruning with Wanda on LLaMA-7B at 70% sparsity · WikiText validation | perplexity: 23.86 -> 22.46 | Helped |
| Add per-matrix refinement to per-block allocation (ablation) | AlphaPruning with SparseGPT on LLaMA-7B at 70% sparsity · WikiText validation | perplexity: 18.54 -> 18.32 | Helped |
| Use logarithmic rather than linear mapping of layer metrics (ablation) | AlphaPruning with Wanda on LLaMA-7B · WikiText validation | perplexity | No clear effect |
| Use linear rather than logarithmic mapping of layer metrics (ablation) | AlphaPruning with SparseGPT on LLaMA-7B · WikiText validation | perplexity | Helped |
Surprising findings
- Compressing more on HTed (heavy-tailed) layers is better for low-rank approximation, opposite to the pruning finding that HTed layers should be pruned less.
- Shape metrics outperform scale metrics (Frobenius and spectral norm) for layer-wise sparsity allocation across LLMs and ViTs.
Limitations
- Evaluation focuses on post-training pruning without fine-tuning; fine-tuning experiments are limited due to compute cost.
- Hyperparameters (s1, s2 / tau) are tuned per model and method via a small sweep.
- Per-matrix allocation is better at sparsity below 50%, so the per-block design is tuned for high-sparsity regimes.
Experimental setup
Datasets: WikiText C4 BoolQ RTE HellaSwag WinoGrande ARC Easy ARC Challenge OpenbookQA ImageNet-1K CIFAR-10
Benchmarks: WikiText BoolQ RTE HellaSwag WinoGrande ARC Easy ARC Challenge OpenbookQA ImageNet-1K
Metrics: perplexity zero-shot accuracy top-1 accuracy decode latency speedup FLOPs
Baselines: Uniform OWL Global thresholding ER ER-Plus LAMP Rank selection Layerwise error thresholding
Builds on: Wanda SparseGPT OWL Heavy-Tailed Self-Regularization theory LAMP
Closest papers by meaning
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.93)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.93)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.93)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.93)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.93)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.