Research map / Large language models / Instruction and fine-tuning
DLP: Dynamic Layerwise Pruning in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DLP, a non-uniform layerwise sparsity method that derives per-layer sparsity from median-based layer unimportance of weight-times-activation scores, enabling high-sparsity LLM pruning without predefined outlier thresholds.
Problem
Uniform layerwise sparsity in LLM pruning degrades badly at high sparsity, and existing non-uniform methods like OWL rely on empirically pre-defined outlier thresholds that are model-dependent.
Approach
Measure each layer's unimportance via the median of Wanda-style scores (|W|*||X||) across the layer, convert to relative importance across layers, and assign sparsity in [R-alpha, R+alpha] so higher-importance layers are pruned less, with no empirically tuned outlier threshold.
Key results
- At 70% sparsity, DLP reduces LLaMA2-7B WikiText perplexity with Wanda from 76.84 (uniform) to 22.79 and improves mean zero-shot accuracy over OWL.
- On DeepSparse with Wanda-pruned LLaMA2-7B-chat, DLP achieves 2.8x speedup at 70% sparsity and 3.5x at 80% sparsity on CPU.
- LoRA fine-tuning of 70%-sparse SparseGPT LLaMA1-7B reduces WikiText perplexity from 17.76 to 12.15.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DLP layerwise sparsity instead of OWL with Wanda | Wanda pruning of LLaMA1-7B · WikiText validation, 70% unstructured sparsity | perplexity: 24.46 -> 20.46 | Helped |
| Use DLP layerwise sparsity instead of OWL with SparseGPT | SparseGPT pruning of LLaMA1-7B · WikiText validation, 70% unstructured sparsity | perplexity: 19.95 -> 17.76 | Helped |
| Use DLP layerwise sparsity instead of OWL with Wanda | Wanda pruning of LLaMA2-7B · WikiText validation, 70% unstructured sparsity | perplexity: 30.58 -> 22.79 | Helped |
| Use DLP layerwise sparsity instead of OWL with Magnitude | Magnitude pruning of LLaMA2-13B · Mean zero-shot accuracy, 70% unstructured sparsity | accuracy: 40.67 -> 44.02 | Helped |
| Increase pruning sparsity with DLP | Wanda-pruned LLaMA2-7B-chat-hf on DeepSparse CPU · End-to-end decoding at 70% sparsity | speedup: 1.0x -> 2.8x | Helped |
| Allocate sparsity per layer rather than per Transformer block (ablation) | DLP + Wanda on LLaMA1-7B · WikiText validation, 70% sparsity | perplexity: 3463.32 -> 20.46 | Helped |
| Prune per output rather than per layer (ablation) | DLP + Wanda on LLaMA1-7B · WikiText validation, 70% sparsity | perplexity: 39.02 -> 20.46 | Helped |
| Add LoRA fine-tuning after pruning (ablation) | DLP + SparseGPT on LLaMA1-7B · WikiText validation, 70% sparsity | perplexity: 17.76 -> 12.15 | Helped |
| Use median rather than mean to measure layer unimportance (ablation) | DLP with SparseGPT · WikiText validation, 70% unstructured sparsity, LLaMA1-7B | perplexity: 18.24 -> 17.76 | Helped |
| Use median rather than mean to measure layer unimportance (ablation) | DLP with Wanda · WikiText validation, 70% unstructured sparsity, LLaMA1-7B | perplexity: 21.03 -> 20.40 | Helped |
| Use median rather than sum to measure layer unimportance (ablation) | DLP with Magnitude · WikiText validation, 70% unstructured sparsity, LLaMA1-7B | perplexity: 3.7e3 -> 3.4e3 | Helped |
Surprising findings
- Per-block sparsity allocation performed worse than per-layer allocation (perplexity 3463.32 vs 20.46 for LLaMA1-7B at 70%).
- Magnitude with DLP on LLaMA1-7B at 70% still yields perplexity in the thousands, showing magnitude pruning remains unstable even with non-uniform sparsity.
Limitations
- Hyperparameter alpha must be tuned per sparsity level via iterative search over a grid.
- Results on high-sparsity Magnitude pruning remain very poor in absolute terms.
- Evaluation mostly on perplexity and seven commonsense tasks; per-block vs per-layer and per-output choices are empirically motivated rather than theoretically justified.
Experimental setup
Datasets: C4 WikiText PTB commonsense170k BoolQ RTE HellaSwag WinoGrande ARC-e ARC-c OpenBookQA
Benchmarks: WikiText PTB C4 BoolQ RTE HellaSwag WinoGrande ARC-e ARC-c OpenBookQA
Metrics: perplexity accuracy end-to-end latency throughput speedup LOD
Baselines: Magnitude SparseGPT Wanda OWL Global ER ER-Plus LAMP AlphaPruning LLM-Pruner OwLore LISA
Builds on: Wanda SparseGPT OWL LLM-Pruner LISA
Closest papers by meaning
- Streamlining Redundant Layers to Compress Large Language Models (ICLR 2025 · similarity 0.92)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.92)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.91)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.91)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.91)
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.