Research map / Architectures and efficiency / Pruning and efficient tuning
LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRAP, a mixed structured compression method for LLMs that applies activation-weighted SVD with per-matrix rank allocation to attention (MHA) sub-layers and gradient-free group channel pruning (keeping the least important 1%) to FFN sub-layers, followed by LoRA recovery.
Problem
LLMs require massive memory and compute; existing structured compression methods treat all Transformer sub-layers identically and often rely on gradient-based importance estimation, which is expensive.
Approach
MHA sub-layers show stronger low-rank structure than FFN sub-layers, so compress MHA with activation-weighted SVD (AWSVD) and allocate parameters across Wq/Wk vs Wv/Wo by their low-rank degree, while compressing FFN with gradient-free channel pruning that retains the least important 1% of weights.
Key results
- At 20% compression on LLaMA-7B without fine-tuning, LoRAP reaches 60.53% average zero-shot accuracy and 15.69 WikiText2 perplexity, versus 56.69% for LLM-Pruner.
- At 50% compression on LLaMA-7B without fine-tuning, LoRAP achieves WikiText2 perplexity 56.96 versus 112.44 for LLM-Pruner.
- At 50% compression with LoRA recovery on LLaMA-7B, LoRAP averages 52.17% accuracy versus 50.39% for LoRAShear and 48.69% for LLM-Pruner.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LoRAP instead of LLM-Pruner at 20% compression without tuning | LLaMA-7B · 20% compression, zero-shot common-sense reasoning | Average accuracy: 56.69 -> 60.53 | Helped |
| use LoRAP instead of LLM-Pruner at 50% compression without tuning | LLaMA-7B · 50% compression, zero-shot common-sense reasoning | Average accuracy: 42.13 -> 47.25 | Helped |
| retain the least-important 1% of FFN weights instead of retaining none (ablation) | LoRAP · 20% compression | WikiText2 perplexity: 16.91 -> 15.69 | Helped |
| retain the least-important 1% of FFN weights instead of retaining none (ablation) | LoRAP · 50% compression | Average accuracy: 46.19 -> 47.25 | Helped |
| retain 2% rather than the least-important 1% of FFN weights (ablation) | LoRAP · 50% compression | Average accuracy: 47.25 -> 46.78 | Hurt |
| allocate MHA parameters at ratio 1:3 rather than 1:1 for (Wq+Wk):(Wv+Wo) (ablation) | LoRAP · LLaMA-7B, 50% compression | Average accuracy: 40.70 -> 47.25 | Helped |
| use L2 rather than L-infinity aggregation for channel importance (ablation) | LoRAP · LLaMA-7B, 50% compression | Average accuracy: 44.09 -> 47.25 | Helped |
| increase the quantity of calibration data (ablation) | LoRAP · WikiText2 and PTB | perplexity | Helped |
| increase calibration sample length (ablation) | LoRAP · WikiText2 and PTB | perplexity | Mixed |
| apply AWSVD rather than AFM to the MHA sub-layer (ablation) | LLaMA-7B at 50% compression · MHA sub-layer, WikiText2 | perplexity: 33.90 -> 19.49 | Helped |
| use channel pruning rather than AWSVD on the FFN sub-layer (ablation) | LLaMA-7B at 50% compression · FFN sub-layer, WikiText2 | perplexity: 225.08 -> 32.64 | Helped |
Surprising findings
- Retaining the least important 1% of weights during FFN channel pruning gives a noticeable perplexity reduction (about 15%) at 50% compression.
- Standard SVD on FFN weights collapses (WikiText2 perplexity 14554), and even activation-weighted SVD performs far worse than channel pruning there.
Limitations
- Performance degrades substantially at compression ratios above 40%, even after LoRA fine-tuning.
- The same compression ratio is applied to all transformer layers, ignoring layer-wise differences.
- Only experiments on LLaMA-family and Vicuna models up to 13B in the main paper.
Experimental setup
Datasets: BookCorpus Alpaca (cleaned) C4 WikiText2 PTB
Benchmarks: WikiText2 PTB BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenbookQA
Metrics: perplexity accuracy MACs latency
Baselines: LLM-Pruner LoRAPrune LoRAShear AFM SVD Head Prune
Builds on: Wanda LLM-Pruner LoSparse FWSVD
Closest papers by meaning
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.88)
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025 · similarity 0.88)
- MoDeGPT: Modular Decomposition for Large Language Model Compression (ICLR 2025 · similarity 0.88)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.88)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.88)
- NOLA: Compressing LoRA using Linear Combination of Random Basis (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.