Research map / Architectures and efficiency / Quantization
SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SVD-LLM, a post-training SVD-based LLM compression method with truncation-aware Cholesky data whitening (so truncated singular values equal compression loss) plus sequential LoRA updates of the low-rank factors.
Problem
SVD-based LLM compression degrades severely at high compression ratios because truncated singular values do not map directly to compression loss and the compressed weights are not updated after truncation.
Approach
Whiten the activations via Cholesky decomposition of XX^T so that the truncated singular values of WS equal the layer's Frobenius compression loss, then recover accuracy by sequentially fine-tuning the two low-rank factors with LoRA.
Key results
- At 40% compression or above on LLaMA-7B, SVD-LLM reduces perplexity by more than 99% on two language modeling datasets versus baselines and achieves over 400% higher average accuracy on six classification datasets.
- SVD-LLM outperforms LLM-Pruner, SliceGPT, and BlockPruner under the same memory budget, with up to 56% perplexity reduction at a 7GB budget on LLaMA-7B.
- SVD-LLM combined with 2-bit QuIP# (after 40% SVD compression) outperforms the training-required 1-bit OneBit without retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ASVD with SVD-LLM | LLaMA-7B compression · WikiText-2, 20% compression | compression time: about $5.5$ hours -> $3.5$ hours; $36\%$ times faster | Helped |
| use SVD-LLM rather than vanilla SVD, FWSVD, or ASVD | LLaMA-7B compression · 10 datasets, compression ratios from 20% to 80% | perplexity and average accuracy: reduces the perplexity by more than 99%; over 400% higher average accuracy | Helped |
| use SVD-LLM instead of the three SVD-based baselines | compressed LLaMA-7B · TruthfulQA and GSM8K, compression ratios 60% and above | generation ability | Helped |
| add truncation-aware data whitening (ablation) | SVD-LLM · LLaMA-7B, WikiText-2, all reported compression ratios | perplexity | Helped |
| add sequential low-rank parameter update (ablation) | SVD-LLM · LLaMA-7B, WikiText-2, all reported compression ratios | perplexity | Helped |
| use both truncation-aware whitening and sequential low-rank parameter update rather than either component alone (ablation) | SVD-LLM · LLaMA-7B, WikiText-2, all reported compression ratios | perplexity | Helped |
| change the number, random seed, or source dataset of calibration data (ablation) | SVD-LLM · LLaMA-7B, 20% compression | final performance: no more than 3% | No clear effect |
| reverse the sequential update order, updating W'_v before W'_u (ablation) | SVD-LLM · LLaMA-7B, WikiText-2, 20% to 80% compression | perplexity | No clear effect |
| replace sequential LoRA fine-tuning with normal LoRA fine-tuning (ablation) | SVD-LLM · reported compression ratios | compression performance | Hurt |
| add sequential LoRA fine-tuning to ASVD rather than use SVD-LLM (ablation) | ASVD · various compression ratios | compression performance | Hurt |
Surprising findings
- Baselines SVD, FWSVD, and ASVD produce zero scores on generation datasets (TruthfulQA, GSM8K) at 60% compression and above, meaning the compressed models lose generation ability.
- Updating the two low-rank factors simultaneously with LoRA is unstable; sequential updating is needed.
Limitations
- Compression accuracy still degrades at high compression ratios.
- Compressing the KV cache adds latency due to recomputing key and value states.
- Compressed models can still produce repetitive or low-quality text even at low compression ratios.
Experimental setup
Datasets: WikiText-2 C4 Alpaca OpenbookQA WinoGrande HellaSwag ARC-e PIQA MathQA TruthfulQA GSM8K
Benchmarks: WikiText-2 C4 OpenbookQA WinoGrande HellaSwag ARC-e PIQA MathQA TruthfulQA GSM8K
Metrics: perplexity accuracy throughput (tokens/sec) peak memory compression time
Baselines: SVD FWSVD ASVD FLAP LLM-Pruner SliceGPT BlockPruner PB-LLM BiLLM OneBit StableLM-3B DRONE
Builds on: ASVD FWSVD DRONE LoRA LLM-Pruner
Closest papers by meaning
- Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives (ICLR 2025 · similarity 0.93)
- LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression (NeurIPS 2026 · similarity 0.91)
- Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression (ICLR 2025 · similarity 0.90)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.89)
- Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models (NeurIPS 2024 · similarity 0.89)
- Compressing Large Language Models by Joint Sparsification and Quantization (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.