Research map / Architectures and efficiency / Pruning and efficient tuning
D2 Prune: Sparsifying Large Language Models via Dual Taylor Expansion and Attention Distribution Awareness
Lang Xiong, Ning Liu, Ao Ren, Yuheng Bai, Haining Fang, Binyan Zhang, Zhe Jiang, Yujuan Tan, Duo Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dual Taylor activation terms (ablation) | SparseGPT · LLaMA-3-8B WikiText2, 50-80% sparsity | perplexity: 8.56 -> 8.48 (50%); 178.01 -> 170.92 (80%) | Improved |
| add dual Taylor activation terms (ablation) | Wanda · LLaMA-3-8B WikiText2, 50-80% sparsity | perplexity: 2245.91 -> 1414.32 (80%) | Improved |
| update q, k, v in all attention layers instead of none (ablation) | D2Prune · LLaMA-2-7B WikiText2, 70% | perplexity: 81.62 -> 23.45 | Improved |
| leave v un-updated while updating q and k (dynamic update) (ablation) | D2Prune · LLaMA-2-7B WikiText2, 70% | perplexity: 23.45 -> 21.10 | Improved |
| leave q or k un-updated instead of v (ablation) | D2Prune · LLaMA-2-7B WikiText2, 70% | perplexity: 21.10 -> 25.16 (q), 24.04 (k) | Hurt |
| update all q/k/v at 80% sparsity vs dynamic (ablation) | D2Prune · LLaMA-2-7B WikiText2, 80% | perplexity: 101.88 -> 92.68 | Improved |
| add LoRA fine-tuning to pruned model (ablation) | D2Prune · LLaMA-2-7B WikiText2, 70% | perplexity: 21.10 -> 9.70 | Improved |
| use optimal q/k/v update configuration across sparsity levels (ablation) | D2Prune · LLaMA-2-7B | perplexity | No clear effect |
| D2Prune versus SparseGPT at 50% on LLaMA-2-7B | D2Prune · LLaMA-2-7B WikiText2, 50% | perplexity: 6.52 -> 6.36 | Improved |
| D2Prune versus Pruner-Zero at 50%, zero-shot | D2Prune · LLaMA-2-7B, 50% | accuracy: 59.26 -> 61.06 | Improved |
| D2Prune versus Wanda at 50% (small gap) | D2Prune · LLaMA-2-7B WikiText2, 50% | perplexity: 6.44 -> 6.36 | No clear effect |
Closest papers by meaning
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.91)
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.90)
- Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.90)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.90)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.90)
- An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
