Research map / Large language models / Language, safety and interpretability
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OWL, a non-uniform layerwise sparsity schedule for one-shot LLM pruning that allocates per-block sparsity according to each block's activation-outlier ratio, plugged into Wanda and SparseGPT.
Problem
One-shot unstructured pruning of LLMs typically uses uniform layerwise sparsity, which underperforms at high sparsity (e.g., 70%), even though non-uniform layerwise sparsity helps in vision models.
Approach
Measure a layerwise outlier distribution (LOD) from Wanda-style outlier scores, then set each Transformer block's sparsity proportional to (1 - outlier ratio), clipped to a window of width lambda around the target sparsity, so layers with more outliers are pruned less.
Key results
- At 70% sparsity on LLaMA-7B, OWL with Wanda reaches WikiText perplexity 24.55 versus 85.77 for uniform Wanda (61.22 lower), and OWL with SparseGPT reaches 19.49 versus 26.30 for SparseGPT.
- OWL with DeepSparse yields 2.6x end-to-end decode speedup at 70% sparsity and 3.9x at 90% on LLaMA-V2-7B-chat (Intel Xeon 8360Y, 36 cores).
- Average zero-shot accuracy over 7 tasks at 70% sparsity on LLaMA-7B rises from 39.39 (Wanda) to 46.47 with OWL, and from 44.93 (SparseGPT) to 48.03 with OWL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace uniform layerwise sparsity with OWL in Wanda pruning | Wanda on LLaMA-7B · WikiText validation, 70% sparsity | perplexity: 85.77 -> 24.55 | Helped |
| Replace uniform layerwise sparsity with OWL in SparseGPT pruning | SparseGPT on LLaMA-7B · WikiText validation, 70% sparsity | perplexity: 26.30 -> 19.49 | Helped |
| Use OWL layerwise sparsity instead of uniform sparsity | Wanda and SparseGPT on LLaMA-V1 · Zero-shot tasks mean accuracy, 70% sparsity | accuracy: 39.39 -> 46.47; 44.93 -> 48.03 | Helped |
| Use OWL with SparseGPT layerwise sparsity | SparseGPT on OPT-6.7B · WikiText validation, 70% sparsity | perplexity: 20.29 -> 22.48 | Hurt |
| Apply OWL layerwise sparsity to Wanda (ablation) | Wanda on LLaMA-7B · WikiText, 70% sparsity | perplexity: 85.77 -> 822.23 | Hurt |
| Use per-block OWL ratios instead of per-layer OWL ratios (ablation) | OWL with Wanda on LLaMA-7B · WikiText, 70% sparsity | perplexity: 86.285 -> 24.55 | Helped |
| Add LoRA fine-tuning (ablation) | OWL w. SparseGPT on LLaMA-7B · WikiText, 70% sparsity | perplexity: 19.49 -> 11.15 | Helped |
| Add LoRA fine-tuning (ablation) | OWL w. SparseGPT on LLaMA-13B · WikiText, 70% sparsity | perplexity: 14.55 -> 9.0 | Helped |
| Replace uniform layerwise sparsity with OWL for mixed N:M pruning | Wanda on LLaMA-7B · WikiText, mixed 3:8 sparsity | perplexity: 42.56 -> 21.49 | Helped |
| Replace uniform layerwise sparsity with OWL in structured pruning | LLM Pruner on LLaMA-7B · WikiText, 70% structured sparsity | perplexity: 90.02 -> 76.99 | Helped |
| Replace uniform layerwise sparsity with OWL | Wanda on ConvNeXt-Base · ImageNet-1K, 70% sparsity | top-1 accuracy: 68.18 -> 68.28 | No clear effect |
| Replace uniform layerwise sparsity with OWL | Wanda on DeiT-Base · ImageNet-1K, 70% sparsity | top-1 accuracy: 49.20 -> 54.24 | Helped |
Surprising findings
- Per-layer OWL assignment performs much worse than per-block assignment (perplexity 86.3 vs 24.55), since it yields nearly uniform sparsity for some projections.
- Inverting the outlier ratio (OWL-inverse) severely degrades perplexity at 60-80% sparsity, supporting that the outlier signal drives the gains.
Limitations
- Gains shrink on larger models (e.g., 6.62 perplexity improvement for LLaMA-65B), and at very high sparsity perplexity remains high (e.g., 1002.87 at 80% for LLaMA-7B).
- Vision-model gains are small and inconsistent (ConvNeXt shows no benefit), with the paper attributing this to weaker outlier phenomena.
- Hyperparameters M and lambda are chosen by small sweeps and per-model settings, without a principled selection rule.
Experimental setup
Datasets: C4 (128 x 2048-token calibration sequences) C4 (30k tokens for LoRA fine-tuning)
Benchmarks: WikiText perplexity BoolQ RTE HellaSwag WinoGrande ARC-e ARC-c OpenbookQA ImageNet-1K
Metrics: perplexity accuracy end-to-end decode latency speedup pruning time
Baselines: Magnitude pruning SparseGPT Wanda Global sparsity Erdős-Rényi (ER) ER-plus LLM Pruner
Builds on: Wanda SparseGPT Erdős-Rényi sparsity LLM.int8 outlier features
Closest papers by meaning
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.92)
- Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.92)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.91)
- Streamlining Redundant Layers to Compress Large Language Models (ICLR 2025 · similarity 0.91)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.90)
- MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.