Research map / Architectures and efficiency / Pruning and efficient tuning
Asymmetry in Low-Rank Adapters of Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows LoRA's B and A matrices play asymmetric roles, and that freezing a random orthogonal A while training only B matches or beats full LoRA with fewer parameters and better generalization bounds.
Problem
LoRA trains both low-rank factors A and B, but their distinct roles and whether both need training are not formally understood; prior work observed the asymmetry only empirically.
Approach
Theory (least squares and logistic regression) shows freezing A (random projection of inputs) is nearly as good as tuning it, while freezing B (projection of outputs) loses expressivity; tuning only B halves parameters and tightens information-theoretic generalization bounds, so it can use a doubled rank at equal budget.
Key results
- On GLUE with RoBERTa-large, random orthogonal frozen A with r=16 (0.8% params) averages 87.5 vs LoRA r=8 at 87.2, matching AdaLoRA (87.9) with fewer parameters.
- Tuning only B consistently beats tuning only A across BART-large XSum/CNN-DailyMail and Llama-2-7B MMLU; e.g., Llama-2 MMLU avg 45.36 for B-tuned r=32 vs 44.51 for A-tuned.
- On DomainBed ViT, freezing random A and tuning B yields better out-of-domain accuracy on several splits than LoRA and generally smaller train-test gaps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Freeze random A and train B instead of freezing random B and training A (ablation) | LoRA on RoBERTa-large · GLUE | Avg. score: 84.2 -> 87.4 | Helped |
| Train B only at rank 16 rather than both A and B at rank 8 (ablation) | LoRA on RoBERTa-large · GLUE | Avg. score: 87.2 -> 87.5 | Helped |
| Freeze random B and train A instead of freezing random A and training B (ablation) | LoRA on Llama-2-7B · MMLU 5-shot | Avg. accuracy: 45.36 -> 44.51 | Hurt |
| Increase the rank of B-only LoRA from 32 to 64 (ablation) | LoRA on Llama-2-7B · MMLU 5-shot | Avg. accuracy: 45.36 -> 46.46 | Helped |
| Freeze random A and train B rather than freeze random B and train A (ablation) | LoRA on ViT · PACS OOD | Accuracy: 66.25 -> 72.55 | Helped |
| Freeze random A and train B rather than freeze random B and train A (ablation) | LoRA on ViT · VLCS OOD | Accuracy: 72.50 -> 75.81 | Helped |
| Freeze random A and train B rather than standard LoRA (ablation) | LoRA on ViT · PACS OOD | Accuracy: 75.58 -> 72.55 | Hurt |
| Use B-only LoRA (r=16) rather than LoRA (r=8) (ablation) | LoRA on ViT · VLCS ID | Accuracy: 73.51 -> 79.10 | Helped |
| Change the initialization of the trainable A in full LoRA from random uniform to random orthonormal (ablation) | LoRA on RoBERTa-large · GLUE | Avg. score: 87.2 -> 87.3 | No clear effect |
Surprising findings
- Freezing A to a random matrix performs nearly as well as tuning it, while reversing roles (tuning only A) substantially hurts performance.
- Tuning only B with doubled rank at matched parameter count can match or exceed standard LoRA, and in some GLUE cases the random-A variant beats AdaLoRA-level results.
Limitations
- Theoretical analysis is for linear or generalized linear models and a single adapted matrix; extension to deep transformers is argued informally.
- Generalization bound is an upper bound and the paper notes lower bounds are hard to obtain, so generalization advantage is not proven tight.
- Reported gains are often within standard error across GLUE tasks, and TerraIncognita showed low-rank adapters struggle to fit.
Experimental setup
Datasets: GLUE XSum CNN/DailyMail Alpaca DomainBed
Benchmarks: GLUE XSum CNN/DailyMail MMLU DomainBed
Metrics: accuracy Matthews correlation Pearson correlation ROUGE-1/2/L in-domain and out-of-domain accuracy
Baselines: LoRA AdaLoRA IA3 LoRA-FA Full fine-tuning Linear probing
Builds on: LoRA LoRA-FA information-theoretic generalization framework
Closest papers by meaning
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.93)
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.92)
- A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models (ICML 2025 · similarity 0.92)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.91)
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.90)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.