Research map / Architectures and efficiency / Pruning and efficient tuning
LSA: Layer-wise Sparsity Allocation for Large Language Model Pruning Based on Minimal Linear Reconstruction Error
Zhiguo Yang, Changjian Deng, Qinke Chen, Zijing Zhou, Jian Cheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LSA allocation instead of uniform sparsity | Wanda on LLaMA1-7B · WikiText, 70% sparsity | perplexity: 83.45 -> 20.66 | Improved |
| use LSA allocation instead of DLP | Wanda on LLaMA2-13B · WikiText, 70% sparsity | perplexity: 16.80 -> 15.36 | Improved |
| use LSA allocation instead of DLP | SparseGPT on LLaMA2-13B · WikiText, 70% sparsity | perplexity: 13.39 -> 12.56 | Improved |
| use LSA allocation instead of OWL/DLP on zero-shot | Wanda on LLaMA1-7B · 7 zero-shot tasks, 70% sparsity | mean accuracy: DLP 44.31 vs LSA 44.19 | No clear effect |
| use LSA instead of uniform for Magnitude pruning | Magnitude on LLaMA2-7B · 7 zero-shot tasks, 70% sparsity | mean accuracy: 33.66 -> 39.17 | Improved |
| add block-wise granularity to LSA | LSA Wanda LLaMA3-8B · WikiText, 70% sparsity | perplexity: 98.75 -> 80.25 | Improved |
| add block-wise granularity to OWL | OWL Wanda LLaMA1-7B · WikiText, 70% sparsity | perplexity: 24.85 -> 57.91 | Hurt |
| add block-wise granularity to DLP | DLP Wanda LLaMA1-7B · WikiText, 70% sparsity | perplexity: 20.89 -> 40.81 | Hurt |
| add block-wise granularity to LSA | LSA Wanda LLaMA1-7B · WikiText, 70% sparsity | perplexity: 20.66 -> 21.60 | Hurt |
| add projection-wise granularity to LSA | LSA Wanda LLaMA1-7B · WikiText, 70% sparsity | perplexity: 20.66 -> 24.82 | Hurt |
| add LoRA fine-tuning to pruned model | LSA Wanda LLaMA1-7B · WikiText, 70% sparsity | perplexity: 20.66 -> 12.27 | Improved |
| add LoRA fine-tuning to pruned model | LSA Wanda LLaMA2-7B · WikiText, 70% sparsity | perplexity: 22.89 -> 11.75 | Improved |
Closest papers by meaning
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.94)
- BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation (ICLR 2024 · similarity 0.94)
- Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.94)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.93)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.92)
- Streamlining Redundant Layers to Compress Large Language Models (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
