Research map / Architectures and efficiency / Pruning and efficient tuning

Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective

ICML 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Layer-wise pruning of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ATP, a theory-motivated layer-wise sparsity allocation for post-training LLM pruning that uses a monotonically increasing arithmetic progression with one tuned common-difference hyperparameter to mitigate "reconstruction error explosion".

Problem

Uniform layer-wise sparsity in post-training LLM pruning ignores layer importance, and existing layer-wise allocation methods are either hand-crafted metrics without theory or costly searches; errors accumulate across layers when sparsifying.

Approach

Theoretically argues reconstruction error propagates and amplifies from earlier to later layers, so earlier layers should get lower sparsity; allocates per-layer sparsity via an increasing arithmetic progression with a single common difference beta found by a small grid search on WikiText-2 perplexity.

Key results

• On 70% sparse LLaMA2-7B with Wanda, ATP reduces WikiText-2 perplexity by 52.10 versus uniform and improves average zero-shot accuracy by 10.50%, outperforming AlphaPruning by 6.71 perplexity and 2.46%.
• ATP reaches 2.63x CPU and 2.23x GPU end-to-end speedup for 70% sparse LLaMA2-7B versus dense.
• Grid search over beta requires only a few attempts (e.g., 9 for 32-layer models with S=0.7, 3 for 70B), taking about 18 minutes, while Bayesian search takes about 33 hours and reaches similar perplexity (22.10 vs 22.16 at 70%).

Measured effects reported in the paper

Surprising findings

• ATP's perplexity matches Bayesian search closely at 50-70% sparsity, suggesting the simple monotone arithmetic progression is near the searched optimum.
• Uniform sparsity at 70% gives perplexity of thousands at 80% sparsity, showing severe degradation that the allocation partly mitigates.

Limitations

• The arithmetic progression form may not be optimal; the authors plan to explore other sparsity schemes.
• A performance gap to lossless sparsification remains, especially at high sparsity.
• The theoretical argument relies on simplifying assumptions (e.g., ignoring cross terms, and an assumed recurrence L_{i+1} = c L_i + f(s_i) with c>1).

Experimental setup

Datasets: WikiText-2 BoolQ ARC Easy ARC Challenge HellaSwag WinoGrande OpenbookQA PIQA GSM8K MMLU Alpaca-GPT4 ImageNet-1K

Benchmarks: WikiText-2 BoolQ ARC Easy ARC Challenge HellaSwag WinoGrande OpenbookQA PIQA GSM8K MMLU ImageNet-1K VQA

Metrics: perplexity zero-shot accuracy 8-shot accuracy 5-shot accuracy Top-1 accuracy speedup (tokens/s)

Baselines: Uniform OWL DSA AlphaPruning ALS Bayesian search Global ER LAMP

Builds on: Wanda SparseGPT OWL AlphaPruning

Closest papers by meaning

• Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.94)
• Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.92)
• The Sparsity Whisperer (NeurIPS 2026 · similarity 0.92)
• Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity (ICML 2024 · similarity 0.92)
• MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models (NeurIPS 2024 · similarity 0.91)
• La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.91)