Research map / Large language models / Language, safety and interpretability
Automatic Pruning Discovery for Large Language Models
Haidong Kang, Lihong Lin, Enneng Yang, Hong-Ning Dai, Hao Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add step-by-step CoT to LLM pruning-rule design (ablation) | AutoPrune search · LLaMA-2 7B WikiText-2 | perplexity: 16.55 -> 7.17 | Improved |
| add graph-driven multi-path CoT over linear CoT (ablation) | AutoPrune search · LLaMA-2 7B WikiText-2 | perplexity: 7.17 -> 6.28 | Improved |
| add SDSA to Wanda (ablation) | Wanda · LLaMA-1 7B WikiText-2, 60% sparsity | perplexity: 10.66 -> 9.88 | Improved |
| add SDSA to SparseGPT (ablation) | SparseGPT · LLaMA-1 7B WikiText-2, 60% sparsity | perplexity: 10.51 -> 8.85 | Improved |
| add SDSA to AutoPrune (ablation) | AutoPrune · LLaMA-2 7B WikiText-2, 60% sparsity | perplexity: 8.93 -> 8.88 | No clear effect |
| replace uniform sparsity with SDSA (ablation) | Wanda · LLaMA-1 7B WikiText-2, 70% sparsity | perplexity: 85.77 -> 66.93 | Improved |
| replace OWL allocation with SDSA (ablation) | Wanda · LLaMA-1 7B WikiText-2, 70% sparsity | perplexity: 75.06 -> 66.93 | Improved |
| use global sparsity allocation instead of uniform (ablation) | Wanda · LLaMA-1 7B WikiText-2 | perplexity: 7.26 -> 14848 at 50% | Hurt |
| vary number of calibration samples 16 to 2048 (ablation) | AutoPrune · LLaMA-1 7B WikiText-2, 50% | perplexity: 7.12 (16) vs 7.11 (2048) | No clear effect |
| use AutoPrune-discovered rule instead of Wanda | Wanda · LLaMA-1 7B WikiText-2, 60% | perplexity: 10.66 -> 9.63 | Improved |
| use AutoPrune rule instead of Wanda | Wanda · LLaMA-2 7B zero-shot, RTE | accuracy: 53.43 -> 58.12 | Improved |
| use AutoPrune rule instead of Wanda | Wanda · LLaMA-2 13B zero-shot, RTE | accuracy: 64.02 -> 61.01 | Hurt |
Closest papers by meaning
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.93)
- Let LLM Tell What to Prune and How Much to Prune (ICML 2025 · similarity 0.93)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.92)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.92)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.92)
- Pruner-Zero: Evolving Symbolic Pruning Metric From Scratch for Large Language Models (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
