Research map / Architectures and efficiency / Pruning and efficient tuning
Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows via infinite-width analysis that non-zero (both A and B) LoRA initialization improves robustness to small learning rates and that subtracting the initial noise from pretrained weights is generally unnecessary.
Problem
Standard LoRA zero-initializes one of its two matrices with no theoretical justification, and fine-tuning is sensitive to learning rate, especially small learning rates that arise under decay.
Approach
Analyze LoRA fine-tuning dynamics with Adam under infinite-width scaling (mu P-style gamma operator) to derive stability, efficiency and robustness conditions; conclude that initializing A and B both to Theta(n^{-1/2}) (Init[AB]) makes LoRA robust to suboptimal learning rates, and that the initial AB noise can be left in the pretrained weights without harming accuracy.
Key results
- On T5-Base QNLI with learning rate 3e-7, Init[AB] accelerates convergence by nearly a factor of two and yields up to 10% accuracy improvement over Init[A] at small learning rates.
- Init[AB] still gives about a 1% accuracy gain over Init[A] when the learning rate is near optimal on GLUE.
- With appropriate initialization variance (beta up to about 4-8), not subtracting the initial AB product from pretrained weights shows no discernible accuracy difference versus Init[AB]; beta >= 8 causes significant drops.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize both A and B with the same non-zero variance instead of zero-initializing B (Init[AB] vs. Init[A]) | T5-Base LoRA · GLUE (MRPC, CoLA, SST-2, QNLI) | test accuracy: up to a 10% accuracy improvement | Helped |
| Initialize both A and B with the same non-zero variance instead of zero-initializing B (Init[AB] vs. Init[A]) | T5-Base LoRA · QNLI, learning rate 3e-7 | convergence speed: nearly a factor of two | Helped |
| Initialize both A and B with the same non-zero variance instead of zero-initializing B (Init[AB] vs. Init[A]) | T5-Base LoRA · GLUE, near-optimal learning rate | test accuracy: 1% accuracy improvement | Helped |
| Increase initialization variance for A while keeping B zero-initialized | T5-Base LoRA · GLUE | test accuracy at smaller learning rates | Helped |
| Initialize both A and B with the same non-zero variance instead of zero-initializing B (Init[AB] vs. Init[A]) | Llama 3-8B LoRA · Commonsense reasoning | average test accuracy | Helped |
| Initialize both A and B with the same non-zero variance instead of zero-initializing B (Init[AB] vs. Init[A]) | Llama 3-8B LoRA · Arithmetic reasoning | average test accuracy | Helped |
| Initialize both A and B with the same non-zero variance instead of zero-initializing B (Init[AB] vs. Init[A]) | LoRA toy model · FashionMNIST test set | test loss | Helped |
| Do not subtract the initialized AB update from pretrained weights (Init[AB+] vs. Init[AB]) (ablation) | LoRA fine-tuning · Suitable initialization variances | fine-tuning accuracy | No clear effect |
| Do not subtract the initialized AB update from pretrained weights (Init[AB+] vs. Init[AB]) (ablation) | LoRA fine-tuning · Initialization variance with β ≥ 8 | fine-tuning accuracy: β ≥ 8 | Hurt |
| Use non-zero initialization in LoRA+ instead of Init[A] | LoRA+ · Various experimental settings | accuracy | Helped |
Surprising findings
- Non-zero initialization of the AB product introduces noise into the pretrained model, yet fine-tuning does not need to start strictly from the pretrained model.
- Un-fine-tuned T5-Base accuracy is unaffected by non-zero LoRA initialization unless the variance is very large.
Limitations
- Theory relies on a single-data-point, single-LoRA-layer setting and asymptotic width scaling with constants ignored.
- Experiments are limited to T5-Base and Llama 3-8B with a fixed rank of 16 and one-epoch training.
Experimental setup
Datasets: GLUE (MRPC) GLUE (CoLA) GLUE (SST-2) GLUE (QNLI) commonsense reasoning benchmark arithmetic reasoning benchmark MNIST FashionMNIST
Benchmarks: GLUE commonsense reasoning arithmetic reasoning
Metrics: test accuracy test loss
Baselines: LoRA (Init[A], zero initialization) LoRA+ PiSSA LoRA-GA LoftQ
Builds on: LoRA LoRA+ Maximal Update Parameterization Kaiming initialization Adam
Closest papers by meaning
- The Impact of Initialization on LoRA Finetuning Dynamics (NeurIPS 2024 · similarity 0.94)
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.91)
- LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.90)
- LoRA vs Full Fine-tuning: An Illusion of Equivalence (NeurIPS 2025 · similarity 0.90)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.90)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.