Research map / Architectures and efficiency / Pruning and efficient tuning
Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that gradient descent on deep overparameterized factorizations has invariant low-dimensional subspaces, enabling exact-ish compression to 2r-width factors, and applies this as "Deep LoRA" for fine-tuning BERT with 3-layer full-width adapters compressed to rank r.
Problem
Overparameterization improves optimization and generalization but raises computational cost; in low-rank learning and LoRA fine-tuning, choosing the rank is hard and overestimating it causes overfitting, especially with limited data.
Approach
Under orthogonal small initialization, GD on each layer of a deep factorization evolves only within an invariant low-dimensional subspace, so the overparameterized factorization can be compressed to 2r-width factors built from the initial gradient; this compression is extended to deep matrix completion and to LoRA as Deep LoRA with three full-width factors per adapted layer.
Key results
- Deep LoRA improves over vanilla LoRA on limited-data GLUE fine-tuning (1024 samples) by +0.028 overall on average, with +0.090 on CoLA and +0.018 on STS-B.
- On few-shot STS-B, Deep LoRA consistently beats vanilla LoRA at n in {16, 64, 256}, with the largest gap at n=16.
- Compressed deep matrix completion trains roughly 5x faster in wall-time than the full-width original while the trajectory aligns with it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace vanilla LoRA with Deep LoRA | BERT-base fine-tuning · GLUE tasks with 1024 samples | validation performance: CoLA +0.090_{\pm 0.002}; MNLI +0.011_{\pm 0.0005}; MRPC +0.0042_{\pm 0.001}; QNLI +0.048_{\pm 0.0009}; QQP +0.005_{\pm 0 | Helped |
| Replace vanilla LoRA with Deep LoRA | T5-base fine-tuning · E2E, 16 samples | BLEU: +0.033 | Helped |
| Replace vanilla LoRA with Deep LoRA | T5-base fine-tuning · E2E, 16 samples | ROUGE: ROUGE-1 +0.032; ROUGE-2 +0.056; ROUGE-L +0.061; ROUGE-LSUM +0.047 | Helped |
| Replace vanilla LoRA with Deep LoRA | T5-base fine-tuning · E2E, 16 samples | METEOR: -0.00036 | Hurt |
| Compress the full-width three-layer factorization (ablation) | Deep LoRA · STS-B, 16 samples | training convergence speed | Helped |
| Use random initialization instead of compression (ablation) | Deep LoRA · STS-B, 16 samples | training convergence speed | Hurt |
| Use compressed wide factors rather than randomly initialized narrow factors (ablation) | Deep matrix completion · 20% observed entries | wall-time to convergence: roughly 5× faster than the original; 3× faster than randomly initialized factorizations of similar width | Helped |
| Set the discrepant update rate to γ = 0 instead of γ = 0.01 (ablation) | Compressed deep matrix completion · 20% observed entries | trajectory alignment | Hurt |
| Increase the LoRA rank r | Deep LoRA · limited-data fine-tuning | generalization error | No clear effect |
| Increase the LoRA rank r | Vanilla LoRA · limited-data fine-tuning | generalization error | Hurt |
Surprising findings
- At 256 samples, most Deep LoRA adapted layers have numerical ranks between 0 and 4, with many layers unchanged from pretrained weights, while vanilla LoRA saturates rank 8 in all layers.
- Direct compression fails for matrix completion because the subspace computed from the masked initial gradient is misaligned, requiring slow updates of the subspace factors.
Limitations
- Theory is proven for deep linear factorization with orthogonal small initialization; the non-linear case is only sketched.
- Deep matrix completion compression requires slowly updating the subspace factors because the initial gradient subspace is perturbed by the mask.
Experimental setup
Datasets: GLUE STS-B E2E
Benchmarks: GLUE STS-B E2E
Metrics: accuracy Pearson correlation BLEU ROUGE METEOR
Baselines: vanilla LoRA
Builds on: LoRA deep matrix factorization implicit regularization Adam
Closest papers by meaning
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.89)
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.89)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.89)
- From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications (ICML 2025 · similarity 0.89)
- Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.89)
- NOLA: Compressing LoRA using Linear Combination of Random Basis (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.