Research map / Architectures and efficiency / Pruning and efficient tuning
Understanding Performance Collapse in Layer-Pruned Large Language Models via Decision Representation Transitions
boyu shi, Chang Liu, Chuanbao Gao, Xu Yang, Xin Geng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove layer 17 (late Silent Phase) instead of layer 19 (ablation) | Llama3-8B · ARC-Challenge | accuracy: 82.68 -> 72.27 | Hurt |
| remove layers 17 and 18 (pre-transition pair) (ablation) | Llama3-8B · 7-task average | accuracy: 75.91 -> 59.24 | Hurt |
| remove layer 17 on HellaSwag (ablation) | Llama3-8B · HellaSwag | accuracy: 71.92 -> 47.39 | Hurt |
| remove layers 17 and 18 on MMLU (ablation) | Llama3-8B · MMLU | accuracy: 64.71 -> 41.42 | Hurt |
| remove decisive-phase layer 19 (ablation) | Llama3-8B · 7-task average | accuracy: 74.83 -> 75.91 | No clear effect |
| remove layer 17 on BoolQ (ablation) | Llama3-8B · BoolQ | accuracy: 77.89 -> 81.22 | No clear effect |
| remove layer 17 on PIQA (ablation) | Llama3-8B · PIQA | accuracy: 80.30 -> 80.58 | No clear effect |
| use Iterative Pruning instead of ShortGPT before SFT at 12.5% ratio | Llama3-8B · 7-task average after SFT | average accuracy: 64.36 -> 83.76 | Improved |
| use Iterative Pruning instead of SLEB at 31.2% ratio | Llama3-8B · 7-task average after SFT | average accuracy: 50.21 -> 84.56 | Improved |
| use Iterative Pruning instead of MKA at 21.9% ratio | Llama3-8B · 7-task average after SFT | average accuracy: 59.53 -> 83.49 | Improved |
| apply SFT to heavily pruned models (40.6%, 43.8%) | Pruned Llama3-8B · ARC-Challenge decision margin | decision transition | No clear effect |
| apply SFT to moderately pruned model (37.5%) | Pruned Llama3-8B · ARC-Challenge decision margin | decision margin | Improved |
Closest papers by meaning
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.88)
- SkipGPT: Each Token is One of a Kind (ICML 2025 · similarity 0.87)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.86)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.86)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.86)
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
