Research map / Architectures and efficiency / Pruning and efficient tuning
OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OATS, a retraining-free compression method that decomposes each transformer weight matrix (after outlier-aware diagonal scaling) into a sparse plus low-rank matrix via alternating thresholding, outperforming SparseGPT, Wanda and DSNoT on LLMs and ViTs.
Problem
Post-training pruning of large transformers suffers steady performance degradation as compression increases, and structured pruning hurts accuracy more than unstructured pruning, while large models are costly to deploy.
Approach
Approximate each weight matrix as a sum of a sparse and a low-rank matrix (Robust PCA) after scaling columns by the square root of the diagonal of the input activation second moment, solved by alternating thresholding (truncated SVD and hard thresholding); Wanda is a special case with rank ratio zero.
Key results
- At 50% compression, OATS improves 5-shot MMLU over the best prior pruning method by 5.42 points on Phi-3 Mini (59.99 vs 54.57 for Wanda).
- On Intel Xeon CPU with DeepSparse, OATS reaches 1.37x speedup over unstructured pruning at 40% compression on Phi-3 Medium (the abstract's figure).
- On ImageNet at 50% compression, DINOv2-Giant with OATS drops only 0.41 points top-1 accuracy relative to dense (86.14 vs 86.55).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OATS sparse-plus-low-rank decomposition instead of the strongest listed pruning baseline | Phi-3 Mini, 50% compression · MMLU 5-shot | accuracy: 54.57 -> 59.99 | Helped |
| Use OATS instead of the strongest listed pruning baseline | Phi-3 Medium, Llama-3 8B, and Llama-3 70B at 50% compression · MMLU 5-shot | accuracy: 69.76 -> 72.28; 53.60 -> 56.46; 72.76 -> 74.79 | Helped |
| Use OATS instead of the strongest listed pruning baseline | Phi-3 Mini, Phi-3 Medium, Llama-3 8B, and Llama-3 70B at 50% compression · Zero-shot task average | accuracy: 66.36 -> 68.41; 73.25 -> 73.39; 64.66 -> 65.71; 73.17 -> 73.30 | Helped |
| Use OATS instead of the strongest listed pruning baseline | Phi-3 Mini, Phi-3 Medium, Llama-3 8B, and Llama-3 70B at 50% compression · WikiText-2 | perplexity: 16.68 -> 15.18; 9.89 -> 9.05; 11.95 -> 10.87; 5.27 -> 4.78 | Helped |
| Use OATS instead of the strongest listed pruning baseline | ViT-Base and DinoV2-Giant at 50% compression · ImageNet validation | top-1 accuracy: 78.44 -> 78.77; 86.04 -> 86.14 | Helped |
| Use OATS instead of unstructured pruning | Phi-3 Medium 15B, single-token CPU inference at 40% compression · CPU inference | speedup: $1.26\times$ -> $1.73\times$ | Helped |
| Add second-moment scaling and row-wise thresholding (ablation) | OATS on Phi-3 Mini at 40% compression, rank ratio 20% · MMLU | accuracy: 62.46 -> 65.84 | Helped |
| Add second-moment scaling and row-wise thresholding (ablation) | OATS on Phi-3 Mini at 40% compression, rank ratio 20% · Zero-shot tasks | accuracy: 67.58 -> 70.71 | Helped |
| Add second-moment scaling and row-wise thresholding (ablation) | OATS on Phi-3 Mini at 40% compression, rank ratio 20% · WikiText-2 | perplexity: 19.21 -> 11.50 | Helped |
| Replace outlier-sensitive D scaling with robust median scaling (ablation) | OATS on Phi-3 Mini at 50% compression · MMLU | accuracy: 59.99 -> 55.54 | Hurt |
| Switch the thresholding order to hard-thresholding first (ablation) | OATS on Phi-3 Mini at 40% compression, rank ratio 20% · MMLU | accuracy: 65.84 -> 65.51 | Hurt |
| Switch the thresholding order to hard-thresholding first (ablation) | OATS on Phi-3 Mini at 40% compression, rank ratio 20% · WikiText-2 | perplexity: 11.50 -> 11.72 | Hurt |
Surprising findings
- Wanda is shown to be a special case of OATS with rank ratio zero, and the authors report that using a median-based robust scaling instead of the second-moment scaling significantly degrades performance.
- At 256-token sequences, OATS and unstructured pruning deliver identical CPU throughput, so the speedup advantage appears mainly for single-token inference.
Limitations
- OATS requires more wall-clock compression time than prior pruning methods (e.g., 80 iterations of alternating thresholding per layer).
- Speedup advantage over unstructured pruning vanishes for longer 256-token sequences in the CPU experiments.
- The paper does not explore alternative robust PCA solvers and the low-rank term adds dense compute that limits GPU gains.
Experimental setup
Datasets: C4 ImageNet ImageNet-21k WikiText-2 MMLU PIQA HellaSwag WinoGrande OpenBookQA RTE BoolQ ARC-e
Benchmarks: MMLU PIQA HellaSwag WinoGrande OpenBookQA RTE BoolQ ARC-e ARC-c WikiText-2 ImageNet
Metrics: 5-shot accuracy zero-shot accuracy perplexity top-1 accuracy throughput (tokens/s) speedup
Baselines: SparseGPT Wanda DSNoT OWL
Builds on: Wanda Robust PCA SparseGPT OWL
Closest papers by meaning
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.88)
- Sparsest Models Elude Pruning: An Exposé of Pruning’s Current Capabilities (ICML 2024 · similarity 0.87)
- GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning (NeurIPS 2024 · similarity 0.87)
- Numerical Pruning for Efficient Autoregressive Models (AAAI 2025 · similarity 0.87)
- Outlier-Efficient Hopfield Layers for Large Transformer-Based Models (ICML 2024 · similarity 0.86)
- Resource-Efficient Transformer Pruning for Finetuning of Large Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.