Research map / Architectures and efficiency / Pruning and efficient tuning
LoRA vs Full Fine-tuning: An Illusion of Equivalence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LoRA-tuned weight matrices develop high-ranking "intruder dimensions" absent in full fine-tuning, and that scaling these down reduces forgetting of pre-training distribution while preserving downstream accuracy.
Problem
Whether LoRA and full fine-tuning learn equivalent solutions even when they reach matched downstream performance, and how their structural differences relate to forgetting and generalization.
Approach
Defines intruder dimensions as top-ranked singular vectors of fine-tuned weights with low cosine similarity to all pre-trained singular vectors, shows LoRA introduces them while full fine-tuning does not, and causally links them to forgetting via singular-value rescaling interventions.
Key results
- Scaling the top intruder dimension by lambda=0.7 on RoBERTa-base fine-tuned on QQP with LoRA r=8 gives 0.0% change in test accuracy and a 33.2% reduction in forgetting.
- Intruder dimension count correlates strongly with forgetting on RoBERTa (Spearman rho=0.971, p<<0.001) but not significantly with test accuracy (rho=-0.3381, p=0.218).
- In continual learning over six RoBERTa tasks, LoRA models degrade faster than full fine-tuning as intruder dimensions accumulate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoRA instead of full fine-tuning | RoBERTa-base fine-tuned on sequence-classification tasks · Pre-training distribution (pseudo-loss) | pseudo-loss (forgetting) | Helped |
| Use LoRA with large learning rates that produce many intruder dimensions | RoBERTa-base, rank 8 · MNLI | pre-training loss (forgetting) | Hurt |
| Set α = 8 rather than α = 2r | LoRA · Fine-tuning tasks and pre-training distribution | generalization (forgetting) | Hurt |
| Scale down top intruder dimensions with λ = 0.7 (ablation) | LoRA rank 8, RoBERTa-base · QQP | pre-training loss change (%): -33.2 | Helped |
| Scale down top intruder dimensions with λ = 0.5 (ablation) | LoRA rank 256, LLaMA2-7B · MetaMath | pre-training loss change (%): -25.2 | Helped |
| Scale down a neighboring pre-trained singular vector instead of an intruder dimension (ablation) | LoRA fine-tuned models · RoBERTa-base tasks | pre-training loss (forgetting) | Hurt |
| Increase the contribution of intruder dimensions (λ > 1) (ablation) | LoRA fine-tuned models · Pre-training distribution | pre-training loss (forgetting) | Hurt |
| Train only B while freezing randomly initialized A (ablation) | LoRA · High-ranking singular vectors | number of intruder dimensions | Helped |
| Increase LoRA rank past a threshold (ablation) | LoRA fine-tuned on LLaMA2-7B math · Top 10 singular vectors | number of intruder dimensions | Helped |
| Use α = 8 instead of α = 2r (ablation) | High-rank LoRA · Fine-tuning update matrices | effective rank | Hurt |
| Sequentially fine-tune across six tasks with LoRA rather than full fine-tuning | RoBERTa-base · Continual learning: MNLI, QQP, SST-2, SIQA, WinoGrande, FEVER | performance on previously learned tasks | Hurt |
| Scale intruder dimensions down with λ = 0.7 (ablation) | LoRA rank 8, RoBERTa-base · QQP | test accuracy change (%): -0.0 | No clear effect |
Surprising findings
- With large learning rates or alpha=8, LoRA can forget more than full fine-tuning despite equal downstream accuracy.
- Scaling down intruder dimensions in LLaMA2-7B MetaMath r=256 (lambda=0.5) increased test accuracy by 1.8% while reducing forgetting by 25.2%.
Limitations
- LLaMA2-7B models were publicly released checkpoints rather than self-trained, limiting control over hyperparameters.
- Intruder dimension definition depends on a similarity threshold epsilon and top-k choice.
- Causal claims rest on rescaling a single top intruder dimension per matrix, and the mechanism for why intruder dimensions arise is only partially explained.
Experimental setup
Datasets: MNLI QQP SST-2 SIQA WinoGrande FEVER MetaMathQA Magicoder-Evol-Instruct Alpaca OpenWebText CCNews Stories
Benchmarks: HumanEval GSM8K HellaSwag WinoGrande ARC-Challenge MNLI SST-2 QQP SIQA FEVER
Metrics: test accuracy pseudo-loss (forgetting) pre-training loss Spearman correlation number of intruder dimensions
Baselines: full fine-tuning AdaLoRA LoRA+ PiSSA VeRA
Builds on: LoRA intrinsic dimension hypothesis PiSSA AdaLoRA LoRA+ VeRA
Closest papers by meaning
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.91)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.91)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.91)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.91)
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.90)
- LoRA Training in the NTK Regime has No Spurious Local Minima (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.