Research map / Optimization / Implicit bias and generalization
Towards Understanding the Dynamics of Low-Rank Adaptation
Shu Ding, Yang Peng, Hangan Zhou, Xinyu Lu, Shangwei Chen, Junhua Huang, Mingxuan Yuan, Wei Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize LoRA A as ETF instead of Gaussian (ablation) | LoRA · GLUE CoLA, RoBERTa rank 8 | accuracy: 57.95 -> 62.18 | Improved |
| ETF init instead of Gaussian (ablation) | ReLoRA · GLUE MNLI, RoBERTa rank 8 | accuracy: 82.79 -> 85.16 | Improved |
| ETF init instead of Gaussian (ablation) | FLORA · GLUE RTE, RoBERTa rank 8 | accuracy: 72.08 -> 74.85 | Improved |
| ETF init instead of default init (ablation) | LoRA-GA and PiSSA · GLUE CoLA, RoBERTa rank 8 | accuracy: 58.70 -> 62.18 (LoRA-GA); 58.14 -> 62.18 (PiSSA) | Improved |
| ETF init instead of Gaussian (ablation) | LoRA · GSM8K, Llama-3.1-8B rank 8 | accuracy: 68.33 -> 71.07 | Improved |
| ETF init instead of Gaussian (ablation) | LoRA · HumanEval, Llama-3.1-8B rank 8 | score: 42.48 -> 46.34 | Improved |
| ETF init instead of Gaussian (ablation) | LoRA · MT-Bench, Mistral-7B-v0.1 rank 8 | score: 6.13 -> 6.53 | Improved |
| ETF init instead of SVD init (ablation) | GaLore · GSM8K, Llama-3.1-8B rank 8 | accuracy: 76.21 -> 78.71 | Improved |
| ETF init instead of SVD init (ablation) | GaLore · GLUE (7 tasks), RoBERTa rank 4/8/16 | accuracy: e.g. QQP rank 4: 91.08 -> 90.15; SST-2 rank 8: 95.15 -> 95.12 | No clear effect |
| ETF init instead of default init (ablation) | LoRA variants · CoLA, MRPC, QNLI, RTE, RoBERTa rank 8 | training loss | Improved |
| ETF init vs Gaussian and orthogonal init for gradient energy preservation (ablation) | rank-4 A on random 100x100 gradient · synthetic gradient, 100 trials | energy preservation ratio | Improved |
| ETF init across ranks 4, 8, 16 (ablation) | LoRA variants · RoBERTa and Llama-3.1-8B | accuracy | Improved |
Closest papers by meaning
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.92)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.92)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.92)
- The Primacy of Magnitude in Low-Rank Adaptation (NeurIPS 2025 · similarity 0.91)
- Flora: Low-Rank Adapters Are Secretly Gradient Compressors (ICML 2024 · similarity 0.91)
- LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
