Research map / Learning theory and algorithms / Sample complexity
The Impact of Initialization on LoRA Finetuning Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that initializing LoRA's A randomly and B to zero (A-init) allows larger stable learning rates and better finetuning than the reverse (B-init), via infinite-width analysis and experiments on RoBERTa, TinyLlama and Llama-7b.
Problem
Which of the two LoRA initialization schemes (random A with zero B, or zero A with random B) should be used, since their product BA is zero at init but their finetuning dynamics may differ; prior work did not study this choice.
Approach
Applies a large-width (mu-P style, gamma-operator) scaling analysis to LoRA to show that random-A/zero-B initialization tolerates learning rates up to Theta(n^-1/2) versus Theta(n^-1) for random-B/zero-A, giving efficient feature learning at the cost of internal feature growth in Z_A.
Key results
- On MNLI with RoBERTa-large, A-init reaches 90.69 best test accuracy vs 89.47 for B-init, with optimal learning rate 8e-5 vs 1e-5.
- Theory predicts maximal stable learning rate Theta(n^-1/2) for A-init versus Theta(n^-1) for B-init; teacher-student experiments show A-init optimal LR larger for widths n >= 2^9.
- On TinyLlama WikiText-2, A-init has larger optimal learning rate and lower test perplexity than B-init; on Llama-7b Flan-v2 and GSM8k, A-init gives generally higher accuracy near the optimal LR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize A randomly and B to zero instead of initializing B randomly and A to zero | LoRA finetuning with RoBERTa-Large · MNLI test | accuracy: 89.47 -> 90.69 | Helped |
| Initialize A randomly and B to zero instead of initializing B randomly and A to zero | LoRA finetuning · GLUE tasks (MNLI, SST2, QNLI) | performance | Helped |
| Initialize A randomly and B to zero instead of initializing B randomly and A to zero | LoRA finetuning with TinyLlama · WikiText-2 | test perplexity | Helped |
| Initialize A randomly and B to zero instead of initializing B randomly and A to zero | LoRA finetuning with Llama-7B · Flan-v2; MMLU 5-shot | accuracy | Helped |
| Initialize A randomly and B to zero instead of initializing B randomly and A to zero | LoRA finetuning with Llama-7B · GSM8K | accuracy | Helped |
| Use init A rather than init B | LoRA finetuning · Large-width limit | maximum stable learning rate: Θ(n^{-1}) -> Θ(n^{-1/2}) | Helped |
| Use init B rather than init A | LoRA finetuning · Large-width limit | efficient feature learning | Hurt |
Surprising findings
- For QNLI, A-init and B-init share the same optimal learning rate even though A-init's optimal accuracy is clearly better.
- Under A-init, the LoRA output Z_B shrinks with width (eta* = o(n^-1/2)), and internal features Z_A grow as Theta(n^1/2), so the favored scheme has internal instability.
Limitations
- Feature learning is defined data-agnostically via magnitude of updates in the infinite-width limit, so no generalization conclusions are drawn.
- Theory uses a single datapoint and a single trainable LoRA layer; results are extended to minibatches heuristically.
- Both schemes are suboptimal asymptotically (internal instability for A-init, undertrained B for B-init), and the gap at finite width is not guaranteed.
Experimental setup
Datasets: GLUE (MNLI, SST2, QNLI) Flan-v2 (100k subset) GSM8k WikiText-2 synthetic teacher-student data
Benchmarks: MNLI SST2 QNLI MMLU GSM8k WikiText-2 perplexity
Metrics: test accuracy optimal learning rate test perplexity train loss
Baselines: LoRA+ (B-init comparison) LoRA default initialization (B zero, A random) from Hu et al.
Builds on: LoRA LoRA+ mu-P Tensor Programs
Closest papers by meaning
- Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics (ICML 2025 · similarity 0.94)
- LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.92)
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.90)
- LoRA vs Full Fine-tuning: An Illusion of Equivalence (NeurIPS 2025 · similarity 0.89)
- The Primacy of Magnitude in Low-Rank Adaptation (NeurIPS 2025 · similarity 0.88)
- Asymmetry in Low-Rank Adapters of Foundation Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.