Research map / Large language models / Language, safety and interpretability
Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes pruning LLM attention weights by directly minimizing the error in the Softmax attention matrix (not linear XW), solved via gradient descent with a convergence guarantee, and reports gains over Wanda and SparseGPT on Llama 3.2-1B perplexity.
Problem
Standard LLM pruning (Wanda, SparseGPT) minimizes error in linear products XW (or separately W_Q, W_K), which poorly approximates the non-linear Softmax attention matrix, degrading performance.
Approach
Learn a single pruning mask on the fused attention weight W = W_Q W_K^T by gradient descent on a loss measuring the Frobenius error between the true and pruned Softmax attention matrices plus L2 regularization, with a PL-inequality-based convergence proof.
Key results
- On Llama 3.2-1B with dense MLP, attention-only pruning at 50% gave PPL 13.885 for the proposed method vs 14.269 (SparseGPT) and 14.912 (Wanda) against dense 12.487.
- With SparseGPT MLP pruning, the proposed attention pruning reached PPL 34.946 vs 36.641 (SparseGPT attn) and 45.435 (Wanda attn).
- Synthetic experiments show lower relative attention approximation error than Wanda and SparseGPT across sequence lengths and pruning ratios.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use attention-matrix approximation pruning instead of SparseGPT for attention | Dense MLP + attention pruning · C4 validation | PPL: 14.269 -> 13.885 | Helped |
| Use attention-matrix approximation pruning instead of Wanda for attention | Dense MLP + attention pruning · C4 validation | PPL: 14.912 -> 13.885 | Helped |
| Use attention-matrix approximation pruning instead of SparseGPT for attention | Wanda MLP + attention pruning · C4 validation | PPL: 26.074 -> 24.427 | Helped |
| Use attention-matrix approximation pruning instead of Wanda for attention | Wanda MLP + attention pruning · C4 validation | PPL: 30.426 -> 24.427 | Helped |
| Use attention-matrix approximation pruning instead of SparseGPT for attention | SparseGPT MLP + attention pruning · C4 validation | PPL: 36.641 -> 34.946 | Helped |
| Use attention-matrix approximation pruning instead of Wanda for attention | SparseGPT MLP + attention pruning · C4 validation | PPL: 45.435 -> 34.946 | Helped |
Limitations
- Experiments are preliminary: attention-only pruning on a single small model (Llama 3.2-1B) with 8 calibration sentences and 128 evaluation sentences.
- Theory relies on assumptions (XX^T ⪰ βI, lower bound δ on softmax entries) that are only empirically checked on C4; regularization/step size must be tuned (lambda, eta).
- Evaluated only perplexity and attention-approximation error, not downstream task accuracy.
Experimental setup
Datasets: C4 synthetic Gaussian matrices
Benchmarks: C4 validation perplexity synthetic relative attention error
Metrics: perplexity relative error
Baselines: SparseGPT Wanda
Builds on: SparseGPT Wanda Polyak-Lojasiewicz inequality gradient descent analysis
Closest papers by meaning
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.92)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.91)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.91)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.91)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.90)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.