Research map / Large language models / Language, safety and interpretability
SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free LLM compression that keeps the top ~15% high-activation-norm FFN neurons intact, low-rank decomposes the rest via SVD-LLM-style whitening, and allocates per-component truncation ranks with a greedy search.
Problem
Compressing billion-parameter LLMs without special hardware or post-training; existing SVD-based low-rank methods lose much accuracy at high compression rates, ignoring activation distribution and differences between FFN and attention components.
Approach
Observes a "soft activation sparsity" in modern LLM FFNs (a small set of neurons with high activation norms dominates), keeps these neurons uncompressed and low-rank decomposes the remaining neurons' weights, plus an adaptive component-wise rank allocation that maximizes retained singular-value energy under a memory budget.
Key results
- At 30% compression on LLaMA-2-70B, SoLA reduces WikiText2 perplexity from 6.95 (SVD-LLM) to 4.44.
- At 20% compression on LLaMA-2-7B, SoLA reaches perplexity 6.52 vs 8.07 for SVD-LLM and average zero-shot accuracy gains of several points on 30% settings.
- Adaptive rank allocation improves perplexity by 8%-18% over uniform allocation across LLaMA-2-7B/13B and Mistral-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SoLA instead of FLAP at 20% compression | LLaMA-2-7B · WikiText2 validation | perplexity: 6.76 -> 6.52 | Helped |
| Use SoLA instead of FLAP at 20% compression | LLaMA-2-13B · WikiText2 validation | perplexity: 5.90 -> 5.61 | Helped |
| Use SoLA instead of SVD-LLM at 30% compression | LLaMA-2-70B · WikiText2 validation | perplexity: 6.95 -> 4.44 | Helped |
| Use SoLA instead of FLAP at 30% compression | Mistral-7B · WikiText2 validation | perplexity: 13.10 -> 7.38 | Helped |
| Use SoLA instead of Bolaco at 20% compression | LLaMA-2-7B · Downstream tasks | average accuracy: 0.5733 -> 0.5692 | Hurt |
| Use SoLA instead of Bolaco at 30% compression | LLaMA-2-13B · Downstream tasks | average accuracy: 0.5608 -> 0.5756 | Helped |
| Use SoLA instead of SVD-LLM at 30% compression | LLaMA-2-70B · Downstream tasks | average accuracy: 0.6091 -> 0.6625 | Helped |
| Use SoLA instead of FLAP at 30% compression | Mistral-7B · Downstream tasks | average accuracy: 0.496 -> 0.517 | Helped |
| Retain 5% prime neurons rather than using 15% prime neurons (ablation) | SoLA on LLaMA-2-13B · WikiText2 test, 20% compression | perplexity: 6.5 -> 5.8 | Helped |
| Remove 1% prime neurons (ablation) | LLaMA-2-13B · WikiText2 test | perplexity: 4.57 -> 9665.4 | Hurt |
| Remove 10% marginal neurons (ablation) | LLaMA-2-13B · WikiText2 test | perplexity: 4.57 -> 4.83 | Hurt |
| Remove 30% marginal neurons (ablation) | LLaMA-2-13B · WikiText2 test | perplexity: 4.57 -> 6.58 | Hurt |
Surprising findings
- Removing the highest-norm neurons in LLaMA-2-13B raises perplexity to ~9665 at only 1% removal, while removing 10% of the lowest-norm neurons barely changes perplexity (4.83 vs 4.57).
Limitations
- Speedup measured only on isolated matrix multiplications, not end-to-end inference.
- Some baselines (LLM-Pruner, Bolaco) are not applicable to GQA models, limiting comparisons on LLaMA-2-70B and Mistral-7B.
- Excludes first and last two layers and v/k projections from compression by heuristic choice.
Experimental setup
Datasets: WikiText2 C4
Benchmarks: WikiText2 MMLU BoolQ PIQA WinoGrande HellaSwag ARC-e ARC-c OBQA
Metrics: perplexity accuracy
Baselines: LLM-Pruner FLAP SliceGPT Bolaco SVD-LLM
Builds on: SVD-LLM ASVD FLAP Wanda
Closest papers by meaning
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.92)
- Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.92)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.92)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.91)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.91)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.