Research map / Architectures and efficiency / Pruning and efficient tuning
The Unreasonable Ineffectiveness of the Deeper Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that open-weight LLMs (2.7B–70B) can have up to roughly half of their deepest layers removed based on angular-distance similarity, with QLoRA healing on C4, retaining QA accuracy until a sharp transition.
Problem
How knowledge is stored in LLM weights and how many layers of pretrained open-weight LLMs can be removed without degrading question-answering performance, with a cheap healing step.
Approach
Prune a contiguous block of n layers chosen by minimal angular distance between the block's input and output representations (on the final token), then heal with a small QLoRA finetune on C4; a simpler variant removes the deepest layers (excluding the final one) with no search.
Key results
- Llama-2-70B retains near-unpruned MMLU and BoolQ accuracy until roughly 40-50% of layers are removed, after which performance transitions sharply to random guessing.
- Healed C4 validation loss increases smoothly and roughly linearly with fraction of layers pruned, with no sharp transition.
- The simple remove-deepest-layers heuristic performs comparably to the similarity-informed strategy after healing, but much worse without healing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prune a similarity-selected layer block and heal with QLoRA | Llama-2, Qwen, Mistral-7B, and Phi-2 models · MMLU | accuracy: 45%-55% pruning fraction for Llama-2; 35% for Mistral 7B; 25% for Phi-2; 20% for Qwen | Mixed |
| Prune a similarity-selected layer block and heal with QLoRA | Evaluated model families · BoolQ | accuracy: 20%-55% pruning fractions | Mixed |
| Add QLoRA healing after layer pruning | Pruned LLMs · MMLU | accuracy | Helped |
| Add QLoRA healing after layer pruning | Pruned LLMs · C4 validation | normalized loss | Helped |
| Prune a similarity-selected layer block without healing | Llama-2-70B · MMLU and BoolQ | accuracy | Hurt |
| Prune a similarity-selected layer block | Llama-2-70B · GSM8K and HellaSwag | accuracy | Hurt |
| Prune a similarity-selected layer block | Llama-2-70B · CoT-MMLU | accuracy | No clear effect |
| Use the simple deepest-layer heuristic instead of similarity-informed block selection (ablation) | Llama-2-70B with healing · MMLU | accuracy | Mixed |
| Use the simple deepest-layer heuristic instead of similarity-informed block selection (ablation) | Llama-2-70B with healing · C4 validation | loss | Hurt |
| Change the ordering of few-shot examples (ablation) | Similarity-informed pruning on Llama-2-13B · MMLU | accuracy | No clear effect |
| Increase the number of few-shot examples (ablation) | Similarity-informed pruning on Llama-2-13B · MMLU | accuracy | Helped |
| Vary the finetuning seed (ablation) | Similarity-informed pruning on Llama-2-13B · MMLU | accuracy | No clear effect |
Surprising findings
- Reasoning tasks GSM8K and HellaSwag degrade with any amount of pruning, while CoT-MMLU stays robust, suggesting deeper layers matter for reasoning.
- Lower LoRA rank (even rank 2) improved MMLU accuracy after healing, which the authors attribute to reduced overfitting to C4.
Limitations
- Evaluation limited to MMLU, BoolQ, and a few reasoning tasks; the paper notes more comprehensive evals are needed.
- Healing uses C4 rather than the models' unknown original pretraining data, and LoRA rank/learning rate were chosen per model with some sweeps.
Experimental setup
Datasets: C4
Benchmarks: MMLU BoolQ C4 validation loss CoT-MMLU GSM8K HellaSwag
Metrics: accuracy normalized cross-entropy loss
Baselines: simple remove-deepest-layers heuristic Block Influence layer pruning (concurrent work)
Builds on: QLoRA LoRA logit lens tuned lens
Closest papers by meaning
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.90)
- Streamlining Redundant Layers to Compress Large Language Models (ICLR 2025 · similarity 0.88)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.88)
- When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs (CVPR 2026 · similarity 0.88)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.88)
- Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.