Research map / Architectures and efficiency / Pruning and efficient tuning
From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies non-uniform low-rank structure of LLM weight matrices via gradient/Hessian subspace stabilization, and proposes WeLore, which uses layer-adaptive SVD compression (LRCs vs N-LRCs) and finetunes only the low-rank components for memory-efficient adaptation.
Problem
Why LLM weight matrices become low-rank and how to set layer-wise rank reduction ratios for compression, plus how to fine-tune memory-efficiently by exploiting which weights are low-rank.
Approach
Links gradient subspace stabilization (via a Hessian spectral gap) to emergence of low-rank weight structure, and uses normalized singular value heavy-tail to assign layer-wise rank reduction, then finetunes only the low-rank components (LRCs) while freezing the rest.
Key results
- At 30% effective rank reduction on LLaMA-2 7B, WeLore-COMP reaches perplexity 14.41 vs 91.99 for uniform reduction on C4 (~6.4x better).
- WeLore-PEFT at 50% compression on LLaMA-2 7B recovers C4 perplexity to 11.87 from 1836.62 without finetuning, outperforming LoRA (21.53 at 50%... see table) and GaLore baselines at 70% compression.
- On downstream tasks at 30% compression, WeLore-PEFT scores 53.33 on SVAMP and 98.67 on CoinFlip versus 40.67 and 75.00 for dense full finetuning on those tasks, with memory close to LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use non-uniform WeLore rank reduction instead of uniform reduction | LLaMa-2 7B compression · C4, 30% rank reduction | perplexity: 91.99 -> 14.41 | Helped |
| use non-uniform WeLore rank reduction instead of OWL reduction | LLaMa-2 7B compression · C4, 20% rank reduction | perplexity: 14.49 -> 8.28 | Helped |
| combine WeLore layer-wise rank allocation with ASVD (ablation) | LLaMa-7B compression · C4, 20% rank reduction | perplexity: 7.75 -> 7.21 | Helped |
| fine-tune LRCs while freezing N-LRCs instead of no fine-tuning (ablation) | WeLore-compressed LLaMa-7B · C4, 50% compression | perplexity: 1836.62 -> 11.87 | Helped |
| fine-tune LRCs while freezing N-LRCs instead of fine-tuning with LoRA | 50%-compressed LLaMa-2 7B · C4 | perplexity: 12.48 -> 11.87 | Helped |
| use WeLore-COMP instead of SVD-LLM compression | LLaMa-7B · Factoid-based QA, 25% compression | score: 71.95 -> 71.89 | Hurt |
| use WeLore-COMP instead of SVD-LLM compression | LLaMa-7B · Multi-turn Conversation, 25% compression | score: 5.79 -> 6.09 | Helped |
| use WeLore-COMP instead of SVD-LLM compression | LLaMa-7B · In-context Summarization, 25% compression | score: 6.12 -> 6.46 | Helped |
| fine-tune LRCs and freeze N-LRCs rather than full fine-tuning | 30%-compressed LLaMa-2 7B · StrategyQA | score: 70.783 -> 69.648 | Hurt |
| fine-tune LRCs and freeze N-LRCs rather than LoRA | 30%-compressed LLaMa-2 7B · StrategyQA | score: 65.255 -> 69.648 | Helped |
Surprising findings
- Finetuning only low-rank components (LRCs) can match or outperform full finetuning on several downstream tasks, despite N-LRCs having ~3x more trainable parameters performing worse.
- Uniform and OWL rank reduction collapse to very high perplexity at moderate compression, while the non-uniform WeLore allocation stays reasonable.
Limitations
- Theoretical results rely on strong assumptions (Lipschitz Hessian, KL condition, uniform spectral gap, reversibility) and the convergence proof only holds for theta > 1/2.
- Compression perplexity degrades sharply at high compression for N-LRCs; WeLore-COMP underperforms SVD-LLM on factoid QA.
- Evaluation of summarization and conversation relies on GPT-4 as judge on small subsets (e.g., 300 articles, 80 MT-Bench questions).
Experimental setup
Datasets: C4 CommonsenseQA SVAMP BoolQ CoinFlip BigBench StrategyQA FreebaseQA CNN/DailyMail MT-Bench
Benchmarks: C4 perplexity CommonsenseQA SVAMP BoolQ CoinFlip BigBench StrategyQA FreebaseQA CNN/DailyMail MT-Bench
Metrics: perplexity accuracy GPT-4 judged score GPU memory throughput
Baselines: Uniform rank reduction OWL SVD-LLM ASVD LoRA GaLore Full finetune
Builds on: LoRA GaLore OWL
Closest papers by meaning
- Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation (ICML 2024 · similarity 0.89)
- Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples (NeurIPS 2025 · similarity 0.89)
- Gradient Weight-normalized Low-rank Projection for Efficient LLM Training (AAAI 2025 · similarity 0.88)
- Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining (NeurIPS 2025 · similarity 0.88)
- COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation (NeurIPS 2025 · similarity 0.88)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.