Research map / Large language models / Language, safety and interpretability
Streamlining Redundant Layers to Compress Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLM-Streamline, which prunes consecutive low-importance layers (by cosine similarity of hidden states) and trains a lightweight network to replace them, plus a new "stability" metric for evaluating pruned classifiers.
Problem
Layer pruning of LLMs removes layers and degrades performance, and accuracy on multiple-choice classification benchmarks can overestimate compressed-model quality because pruned models guess correctly on uncertain samples.
Approach
Identify contiguous layers with highest input-output cosine similarity to prune, then train a lightweight network (FFN, SwiGLU, or Transformer layer) with MSE on hidden states to replace them; additionally introduce a stability metric weighting prediction consistency before/after pruning by the original model's PPL-based confidence (std via exp weighting).
Key results
- On Llama2-7B at ~25% pruning, Ours (FFN) retains 93.1% of classification performance (average accuracy 45.8) versus 78.0% for LLM-Pruner and 81.9% for LaCo.
- On generation benchmarks, Ours (Layer) retains 77.2% of Llama2-7B performance versus 63.8% for LLM-Pruner.
- Layer replacement (FFN) achieves 46.3 accuracy and 85.2 stability versus LoRA's 44.5 and 82.1 while using about 27.0G vs 56.4G GPU memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add an FFN replacement network after layer pruning (ablation) | LLM-Streamline on Llama2-7B · Classification benchmarks (average) | accuracy: 44.0 -> 45.8 | Helped |
| add a lightweight replacement network after layer pruning (ablation) | LLM-Streamline on Llama2-13B · Generation benchmarks (average) | accuracy / ROUGE1 average: 22.0 -> 28.4 | Helped |
| use FFN rather than SwiGLU as the replacement network (ablation) | LLM-Streamline on Llama2-7B · Classification benchmarks (average) | accuracy: 44.2 -> 45.8 | Helped |
| halve the FFN replacement network's intermediate size (ablation) | LLM-Streamline on Llama2-7B · Classification benchmarks (average) | accuracy: 45.8 -> 46.0 | Helped |
| initialize the replacement Transformer layer from the last pruned layer rather than averaging pruned-layer weights (ablation) | LLM-Streamline on Llama2-7B · Classification benchmarks (average) | accuracy: 44.4 -> 45.6 | Helped |
| replace layer replacement with LoRA rank 128 | LLM-Streamline on Llama2-7B · Classification benchmarks (average) | accuracy: 46.8 -> 44.5 | Hurt |
| replace layer replacement with LoRA rank 128 | LLM-Streamline on Llama2-7B · Classification benchmarks (average) | stability: 85.7 -> 82.1 | Hurt |
| prune using perplexity rather than cosine similarity (ablation) | Llama2-7B pruning · Classification and generation benchmarks (average) | average: 39.1 -> 35.0 | Hurt |
| use cosine similarity rather than perplexity to select pruned layers (ablation) | Llama2-7B pruning · SlimPajama | perplexity: 12.1 -> 19.7 | Hurt |
| add a Transformer-layer replacement network (ablation) | LLM-Streamline on Llama2-7B · Inference (sequence length 128) | tokens/s: 25.68 -> 25.68 | No clear effect |
| use layer replacement rather than direct layer pruning (ablation) | LLM-Streamline on Llama2-7B · GSM8K | accuracy: 1.97 -> 1.82 | Hurt |
| use a Transformer-layer replacement rather than no replacement (ablation) | LLM-Streamline on OPT-6.7B · Generation benchmarks (average) | accuracy / ROUGE1 average: 1.6 -> 21.8 | Helped |
Surprising findings
- Pruned models often gain accuracy on Race-M and Race-H after pruning, suggesting accuracy overestimates compressed-model quality.
- Without any lightweight network, Llama2-13B pruned by ShortGPT-style removal achieves the highest classification accuracy among methods but lower stability and generation performance.
Limitations
- Inference speedup is slightly inferior to SliceGPT and LLM-Pruner (Appendix inference speed).
- Transformer-layer replacement had not converged after 20 epochs, requiring more compute to realize its potential.
- Generation performance (e.g., GSM8K) is near zero for most pruning methods, and the stability metric relies on the original model's PPL-based confidence.
Experimental setup
Datasets: SlimPajama-6B SlimPajama
Benchmarks: CMNLI HellaSwag PIQA CHID WSC CommonSenseQA BoolQ MMLU CMMLU Race-M Race-H C3
Metrics: accuracy stability ROUGE-1 retained performance
Baselines: LLM-Pruner SliceGPT LaCo ShortGPT (Ours None) SLEB UIDL LoRA
Builds on: ShortGPT LaCo LoRA Sheared LLaMA
Closest papers by meaning
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.93)
- DLP: Dynamic Layerwise Pruning in Large Language Models (ICML 2025 · similarity 0.92)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.92)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.91)
- Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity (ICML 2024 · similarity 0.91)
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.