Research map / Architectures and efficiency / Pruning and efficient tuning
BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BESA, a differentiable blockwise pruning method that learns layer-specific sparsity allocation via parameter-efficient probability masks, pruning LLaMA-family models to 50% unstructured sparsity on a single A100.
Problem
Post-training weight pruning of LLMs with layer-wise reconstruction causes error accumulation and requires hand-tuned uniform per-layer sparsity, degrading performance.
Approach
Minimize block-wise (transformer block) reconstruction error while learning per-layer (or per-row) sparsity differentiably: weights are sorted by Wanda importance, and a small set of learnable coefficients over candidate pruning rates generates straight-through binary masks, so sparsity is optimized by gradient descent.
Key results
- At 50% unstructured sparsity, BESA reaches WikiText2 perplexity 6.86 on LLaMA-7B versus 7.22 for SparseGPT and 7.26 for Wanda.
- On LLaMA2-70B, BESA reports WikiText2 perplexity 4.09 versus 4.25 for SparseGPT and 4.22 for Wanda.
- Joint pruning and 4-bit OmniQuant quantization gives LLaMA-7B WikiText2 perplexity 7.00, outperforming Wanda applied to the quantized model (7.44).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use blockwise parameter-efficient sparsity allocation instead of layerwise pruning | 50% unstructured pruning of LLaMA models · WikiText2 | perplexity: 7.22 -> 6.86 (LLaMA-7B, SparseGPT); 7.26 -> 6.86 (LLaMA-7B, Wanda) | Helped |
| Use blockwise parameter-efficient sparsity allocation instead of layerwise pruning | 50% unstructured pruning of LLaMA models · C4 | perplexity: 9.31 -> 8.96 (LLaMA-7B, SparseGPT); 9.34 -> 8.96 (LLaMA-7B, Wanda) | Helped |
| Use blockwise parameter-efficient sparsity allocation instead of layerwise pruning | 50% unstructured pruning of LLaMA models · PTB | perplexity: 79.25 -> 66.96 (LLaMA-7B, SparseGPT); 22.67 -> 22.87 (LLaMA2-70B, SparseGPT) | Mixed |
| Train for more epochs rather than the 1-epoch default (ablation) | BESA on LLaMA-7B · WikiText2 | perplexity: 6.86 -> 6.85 (3 epochs); 6.86 -> 6.84 (10 epochs); 6.86 -> 6.86 (30 epochs) | No clear effect |
| Train for more epochs rather than the 1-epoch default (ablation) | BESA on LLaMA-7B · PTB | perplexity: 66.96 -> 67.37 (3 epochs); 66.96 -> 66.83 (10 epochs); 66.96 -> 67.09 (30 epochs) | Mixed |
| Use sparsity step 0.1 instead of 0.01 (ablation) | BESA on LLaMA-7B · PTB | perplexity: 66.96 -> 69.29 | Hurt |
| Use sparsity step 0.001 instead of 0.01 (ablation) | BESA on LLaMA-7B · PTB | perplexity: 66.96 -> 66.52 | Helped |
| Replace Wanda importance metric with weight magnitude (ablation) | BESA on LLaMA-7B · WikiText2 | perplexity: 6.86 -> 7.43 | Hurt |
| Replace Wanda importance metric with SparseGPT importance metric (ablation) | BESA on LLaMA-7B · WikiText2 | perplexity: 6.86 -> 8.73 | Hurt |
Surprising findings
- BESA's zero-shot average accuracy on LLaMA-65B (73.73) slightly exceeds the dense model (73.71).
- Using a 0.1 sparsity step gives a lower WikiText2 perplexity than 0.01, but the authors chose 0.01 because it yields better C4 perplexity and easier convergence.
Limitations
- Row-wise sparsity produces unstructured masks that are not directly parallelizable and need a custom CUDA operator.
- Speedup is evaluated only in the ViTCoD accelerator simulator, not on real GPU hardware, since BESA's non-uniform sparsity does not fit n:m patterns.
- Zero-shot results show more variance than perplexity, and performance drops noticeably at high sparsity on some models.
Experimental setup
Datasets: C4
Benchmarks: WikiText2 C4 PTB PIQA BoolQ HellaSwag WinoGrande ARC-e ARC-c
Metrics: perplexity zero-shot accuracy average accuracy simulated runtime cycles and speedup
Baselines: SparseGPT Wanda
Builds on: Wanda SparseGPT OmniQuant
Closest papers by meaning
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.92)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.92)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.91)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.91)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.91)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.