Research map / Architectures and efficiency / Pruning and efficient tuning
Olica: Efficient Structured Pruning of Large Language Models without Retraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Olica prunes LLM attention and FFN layers without any retraining, using orthogonal neuron decomposition (PCA on W_qk and W_vo with a fast one-side SVD) plus ridge-regression linear calibration with low-rank residual correction.
Problem
Structured pruning of LLMs typically needs expensive retraining (LoRA on tens of thousands of instruction samples) to recover accuracy lost from corrupted inter-layer correlations; this paper seeks pruning that needs no retraining and little data.
Approach
Treat the MHA products W_q W_k^T and W_v W_o^T as unified entities and apply PCA via SVD (orthogonal neuron decomposition) so pruning preserves the original structure; a fast variant runs SVD on only one matrix per pair, and FFN residual errors are reconstructed with a closed-form ridge regression whose solution is low-rank SVD-compressed, with layers chosen by an MC2 linear-recoverability criteri
Key results
- On LLaMA-7B at 20% sparsity, Olica reaches 64.54% average zero-shot accuracy and WikiText-2 PPL 15.35, versus 63.81% and 16.99 for SlimGPT (no retraining for Olica).
- At 33% sparsity on LLaMA-7B, Olica achieves PPL 19.83 and 61.21% average accuracy, while using 256 calibration samples and about 7 minutes of pruning on one A100 80GB.
- Fast-OND cuts pruning runtime from 2910s (OND) to 413s on LLaMA-7B with comparable PPL (15.35 vs 15.17).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Olica instead of LoRAP | LLaMA-7B · 25% sparsity | average accuracy: 62.57 -> 63.54 | Helped |
| use Olica instead of LoRAP | LLaMA-7B · 33% sparsity | WikiText2 perplexity: 21.66 -> 19.83 | Helped |
| use Olica instead of LoRAP | LLaMA-13B · 20% sparsity | average accuracy: 66.84 -> 67.67 | Helped |
| use Olica instead of LoRAP | LLaMA-2-7B · 30% sparsity | average accuracy: 58.89 -> 61.14 | Helped |
| use Olica instead of LoRAP | Vicuna-7B · 20% sparsity | average accuracy: 64.39 -> 64.88 | Helped |
| add linear calibration (ablation) | Fast-OND on LLaMA-7B · 33% sparsity | WikiText2 perplexity: 20.34 -> 19.83 | Helped |
| add linear calibration (ablation) | Fast-OND on LLaMA-7B · 33% sparsity | mean accuracy: 60.68 -> 61.21 | Helped |
| replace standard OND with Fast-OND (ablation) | LLaMA-7B · 20% sparsity | runtime: 2910s -> 413s | Helped |
| replace standard OND with Fast-OND (ablation) | LLaMA-7B · 20% sparsity | WikiText2 perplexity: 15.17 -> 15.35 | Mixed |
| replace Fast-OND with standard SVD (ablation) | LLaMA-7B · without linear calibration | WikiText2 perplexity: 20.34 -> 71.01 | Hurt |
| replace Fast-OND with Activation-Weighted SVD (ablation) | LLaMA-7B · without linear calibration | WikiText2 perplexity: 20.34 -> 25.78 | Hurt |
| replace Fast-OND with Wanda (ablation) | LLaMA-7B · without linear calibration | WikiText2 perplexity: 20.34 -> 20.94 | Hurt |
Surprising findings
- At 33% sparsity on LLaMA-7B, SlimGPT achieves higher average accuracy (61.41 on SlimGPT row vs 61.21 for Olica in the table), so Olica is not uniformly best on accuracy.
- Plain SVD without importance-based selection collapses (PPL 71.01 without calibration), showing that the choice of decomposition matters greatly.
Limitations
- Performance degrades sharply at high sparsity (50%) and for LLaMA-30B beyond 40% sparsity without retraining.
- Relies on approximations for RoPE models (PCA applied separately to W_q and W_k) and on heuristic layer selection and sparsity allocation.
Experimental setup
Datasets: BookCorpus Alpaca WikiText-2 BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenBookQA C4
Benchmarks: WikiText-2 BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenBookQA
Metrics: perplexity zero-shot accuracy average accuracy parameters MACs memory latency runtime
Baselines: LLM-Pruner SlimGPT LoRAP LoraPrune Compresso FLAP SliceGPT LLM-Surgeon DISP-LLM SVD AWSVD Wanda
Builds on: Wanda LoRAP SliceGPT LLM-Pruner
Closest papers by meaning
- SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.92)
- Fluctuation-Based Adaptive Structured Pruning for Large Language Models (AAAI 2024 · similarity 0.91)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.90)
- DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models (NeurIPS 2024 · similarity 0.90)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.90)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.