Research map / Architectures and efficiency / Pruning and efficient tuning
Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoMe, a progressive layer-pruning framework that merges adjacent layers by concatenating high-sensitivity channels instead of linear weight averaging, followed by hierarchical layer-wise distillation for post-training recovery.
Problem
Structured layer pruning of LLMs causes notable performance degradation, linear weight-sum layer merging over-smooths FFN weights, and pruning methods lack effective post-training recovery.
Approach
Channel sensitivity (activation magnitude times weight norm) selects the most important channels from adjacent layers, which are concatenated into a merged layer that keeps the hidden size; SBI guides which layer groups to merge, and a hierarchical KL-based distillation aligns each merged layer with its original layer group (CoMe-mp per-layer or CoMe-sp joint).
Key results
- Pruning 30% of LLaMA-2-7b parameters, CoMe retains 83% of original average accuracy after post-training (abstract).
- CoMe-sp surpasses LLM-Streamline by 2.4% average accuracy on LLaMA-2-7b after post-training and by 5.2% on Qwen-3-4b (as stated in text).
- Removing concatenation-based merge from CoMe lowers average accuracy by 2.12% and RP by 3.53%, and raises Wiki-2 PPL by 4.73.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly optimize mapped fusion layers in one process rather than sequentially (ablation) | CoMe post-training · LLaMA-2-7B, 30% pruning | Avg accuracy: 50.76 -> 52.58 | Helped |
| jointly optimize mapped fusion layers in one process rather than sequentially (ablation) | CoMe post-training · Qwen3-4B, 30% pruning | Avg accuracy: 45.28 -> 47.22 | Helped |
| replace concatenation-based merging (ablation) | CoMe · LLaMA-2-7B, 30% pruning | Avg accuracy: 48.17 -> 46.05 | Hurt |
| replace concatenation-based merging (ablation) | CoMe · LLaMA-2-7B, 30% pruning | C4 perplexity: 19.93 -> 25.48 | Hurt |
| replace concatenation-based merging (ablation) | CoMe · LLaMA-2-7B, 30% pruning | Wiki-2 perplexity: 16.53 -> 21.26 | Hurt |
| increase the merge-ratio exponent p above 2 (ablation) | CoMe · LLaMA-2-7B pruning | Avg accuracy: average accuracy exceeding 48% | Hurt |
| merge more than two layers at a time (ablation) | CoMe · LLaMA-2-7B pruning | Avg accuracy: over 3% | Hurt |
| use QA calibration data such as MMLU instead of pre-training-style data (ablation) | CoMe channel-importance calibration · LLaMA-2-7B pruning | Perplexity: over 25% | Hurt |
| use fewer than eight calibration samples (ablation) | CoMe channel-importance calibration · LLaMA-2-7B pruning | Perplexity | Hurt |
| remove the merge step (ablation) | MKA · LLaMA-2-7B, 30% pruning | Avg accuracy: 46.36 -> 46.24 | No clear effect |
| remove the merge step (ablation) | LaCo · LLaMA-2-7B, 30% pruning | Avg accuracy: 44.71 -> 45.09 | Mixed |
| remove the merge step (ablation) | FuseGPT · LLaMA-2-7B, 30% pruning | Avg accuracy: 45.88 -> 45.21 | Hurt |
Surprising findings
- Linear weight-sum layer merging (WSLP) yields negligible gains over direct layer pruning, changing average accuracy by less than 0.7%.
- Calibration data distribution changes PPL by over 25% (using MMLU) but affects downstream average accuracy by under 1.5%.
Limitations
- Uses a heuristic, uniform parameter preservation ratio for all Transformer components, limiting adaptability across architectures.
- Merging more than two layers per step degrades performance due to parameter distribution differences.
- Posterior-based variant search grows exponentially when merging more layers per iteration.
Experimental setup
Datasets: WikiText-2 C4 PG19 MMLU
Benchmarks: ARC-challenge ARC-easy HellaSwag OpenBookQA PIQA WinoGrande MMLU C4 perplexity WikiText-2 perplexity
Metrics: average accuracy retained performance perplexity
Baselines: Magnitude+ Taylor+ ShortGPT SLEB FuseGPT-MI FuseGPT-MI-F MKA LaCo LLM-Streamline
Builds on: ShortGPT LaCo FuseGPT LLM-Streamline MKA
Closest papers by meaning
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.93)
- Streamlining Redundant Layers to Compress Large Language Models (ICLR 2025 · similarity 0.93)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.93)
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.93)
- A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024 · similarity 0.92)
- Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.