Research map / Architectures and efficiency / Pruning and efficient tuning
Demystifying When Pruning Works via Representation Hierarchies
Shwai He, Guoheng Sun, Haichao Zhang, Yun Fu, Ang Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| drop 8 attention layers | Mistral-7B-Instruct · multiple-choice average | accuracy (%): 69.3 -> 69.8 | No clear effect |
| drop 8 MLP layers | Mistral-7B-Instruct · multiple-choice average | accuracy (%): 69.3 -> 64.3 | Hurt |
| drop 8 attention layers | Mistral-7B-Instruct · generation average | score: 22.3 -> 13.2 | Hurt |
| drop 8 MLP layers | Mistral-7B-Instruct · generation average | score: 22.3 -> 0.8 | Hurt |
| drop 8 attention layers | Mistral-7B-Instruct · GSM8K | accuracy (%): 48.4 -> 36.2 | Hurt |
| drop 8 attention layers | E5-Mistral · BEIR retrieval average | nDCG@10: 58.9 -> 53.4 | Hurt |
| drop 8 MLP layers | E5-Mistral · BEIR retrieval average | nDCG@10: 58.9 -> 56.8 | Hurt |
| drop 8 MLP layers | E5-Mistral · TREC-COVID | nDCG@10: 85.9 -> 79.6 | Hurt |
| apply intra-layer pruning (Wanda 50%, 4:8, 2:4) | Qwen2.5-7B, Llama3-8B, Mistral-7B · GSM8K vs HellaSwag | performance (%) | Mixed |
| measure deviation in logit space versus embedding space after layer drop | Qwen-2.5-7B-Instruct · cosine similarity | 1 - CosineSim | No clear effect |
| map logits to probabilities with softmax | Qwen-2.5-7B-Instruct · probability space deviation | cosine similarity | Hurt |
| evaluate over generation steps rather than first step | Qwen-2.5-7B-Instruct Drop-8A · representation similarity | cosine similarity | Hurt |
Closest papers by meaning
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.88)
- Pruner-Zero: Evolving Symbolic Pruning Metric From Scratch for Large Language Models (ICML 2024 · similarity 0.88)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.88)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.88)
- Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing (ICLR 2025 · similarity 0.88)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
