Research map / Large language models / Language, safety and interpretability
Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptive Layer Sparsity (ALS), which allocates per-layer sparsity for LLM pruning by solving a linear program that maximizes inter-layer independence (a kernel-based redundancy metric), and plugs it into Wanda, SparseGPT, and Magnitude pruning.
Problem
Uniform layer-wise sparsity in LLM pruning ignores varying layer importance and yields suboptimal performance at high sparsity; prior non-uniform methods (BESA, OWL) are costly or rely on hand-tuned hyperparameters.
Approach
Measure inter-layer redundancy with a Cauchy-Schwarz-bounded Redundancy Metric derived from a kernel-density approximation of mutual information over calibration activations, then formulate per-layer sparsity allocation as a linear program that assigns higher sparsity to more redundant layers under a model-size budget.
Key results
- At 50% sparsity on LLaMA-V2 7B, Wanda w. ALS reduces WikiText-2 perplexity from 12.31 to 11.61 and average zero-shot accuracy from 61.88% to 62.84%.
- On LLaMA-V3 8B at 50% sparsity, Magnitude w. ALS improves average accuracy over Magnitude by about 14.14 points (43.29% to 57.43%).
- On LLaMA-V2 13B at 50% sparsity, Wanda w. ALS achieves perplexity 9.86 versus 10.11 for Wanda w. OWL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ALS to Magnitude pruning | Magnitude on LLaMA-V2-13B · WikiText-2, 50% sparsity | perplexity: 11.59 -> 10.78 | Helped |
| Apply ALS to SparseGPT pruning | SparseGPT on LLaMA-V2-13B · WikiText-2, 50% sparsity | perplexity: 11.32 -> 9.44 | Helped |
| Apply ALS to Wanda pruning | Wanda on LLaMA-V2-13B · WikiText-2, 50% sparsity | perplexity: 11.21 -> 9.86 | Helped |
| Apply ALS to Magnitude pruning | Magnitude on LLaMA-V3-8B · WikiText-2, 50% sparsity | perplexity: 1.1e3 -> 30.20 | Helped |
| Apply ALS to SparseGPT pruning | SparseGPT on LLaMA-V3-8B · WikiText-2, 50% sparsity | perplexity: 16.87 -> 10.23 | Helped |
| Apply ALS to Wanda pruning | Wanda on LLaMA-V3-8B · WikiText-2, 50% sparsity | perplexity: 15.01 -> 12.30 | Helped |
| Apply ALS to pruning | Magnitude, SparseGPT, and Wanda on LLaMA-V3-8B · Zero-shot task mean, 50% sparsity | accuracy: 43.29 -> 57.43; 54.18 -> 64.00; 58.12 -> 62.48 | Helped |
| Apply ALS to SparseGPT pruning | SparseGPT on LLaMA-V1-13B · WikiText-2, 50% sparsity | perplexity: 9.90 -> 9.97 | Hurt |
| Apply ALS to Wanda pruning | Wanda on LLaMA-V2-7B · WikiText-2, 70% sparsity | perplexity: 219.46 -> 214.10 | Helped |
| Apply LoRA fine-tuning after Wanda pruning with ALS (ablation) | Wanda with ALS on LLaMA-V1-7B · WikiText-2, 50% sparsity | perplexity: 12.47 -> 7.65 | Helped |
| Apply LoRA fine-tuning after Wanda pruning with ALS (ablation) | Wanda with ALS on LLaMA-V1-7B · Zero-shot task mean, 50% sparsity | accuracy: 61.47 -> 71.64 | Helped |
| Use output features instead of input features (ablation) | ALS on LLaMA2-13B · WikiText2 | perplexity: 10.070 -> 10.012 | Helped |
Surprising findings
- ALS slightly worsens SparseGPT perplexity on LLaMA-V1 13B (9.90 to 9.97), a case where it does not help.
- Increasing calibration samples from 2 to 256 reduces perplexity by only 0.11, indicating robustness to calibration size.
Limitations
- Including intra-layer component-level allocation increases the redundancy matrix size and computation by roughly 49x, as the authors note in their appendix.
- Gains are inconsistent: ALS slightly hurts some settings (e.g., SparseGPT on LLaMA-V1 13B) and the 70% sparsity results often collapse to very high perplexity.
Experimental setup
Datasets: C4
Benchmarks: WikiText-2 Winogrande PIQA OpenBookQA HellaSwag BoolQ ARC-Easy ARC-Challenge RTE
Metrics: perplexity accuracy
Baselines: Magnitude SparseGPT Wanda OWL BESA
Builds on: Wanda SparseGPT OWL BESA
Closest papers by meaning
- Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.94)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.93)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.92)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.92)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.92)
- Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.