Research map / Architectures and efficiency / Pruning and efficient tuning
The Expressive Power of Low-Rank Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis of the expressive power of LoRA showing that low-rank adapters on a frozen FNN or Transformer can exactly represent a target model of comparable size, with rank thresholds set by width, depth and the error matrix rank.
Problem
Understanding what LoRA can represent theoretically: the minimal adapter rank needed for a frozen (possibly random) network to exactly match a target network, the approximation error at lower ranks, and how depth and width affect this.
Approach
Decomposes the change in a product of LoRA-adapted linear layers into a sum of L rank-R terms, so the product can match the best rank-(RL) approximation of the error matrix; extends this to ReLU FNNs via layer partitioning and to Transformers by tuning attention weights, with explicit adapter constructions.
Key results
- For linear and FNN experiments with D=16, MSE approaches zero when LoRA-rank reaches about 8 (= D/(L/L_tgt)), matching the theoretical threshold.
- LoRA with rank at least D/M (M = L/L_tgt) can exactly adapt a random frozen FNN into a target FNN, using about 2D^2 L_tgt learnable parameters, nearly optimal up to a factor of 2.
- Tuning the last L-1 layers of a random FNN cannot match even a one-layer target FNN, while LoRA can when L >= 3 with no more parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoRA rank R ≥ 8 | LoRA adapter construction · Linear model approximation and FNN approximation | MSE: R ≥ 8 | Helped |
| Pretrain the frozen model on the target distribution | LoRA · Linear model approximation and FNN approximation | LoRA rank required for near-zero MSE | Helped |
| Use the construction-derived adapters instead of gradient-update adapters | LoRA · Linear model approximation | MSE | No clear effect |
| Use gradient-update adapters instead of the construction-derived adapters | LoRA · FNN approximation at small rank | MSE | Helped |
| Use a larger model for a fixed downstream task | LoRA · Fixed downstream task | LoRA rank required for desired performance | Helped |
| Use a frozen model closer to the target model | LoRA · Target-model approximation | LoRA rank required for desired performance | Helped |
| Update bias parameters in addition to weight matrices | LoRA · FNN approximation | Approximation performance | Helped |
| Apply LoRA to attention weights | Transformer networks · Transformer networks | Approximation performance | Helped |
| Use LoRA instead of final-layer tuning | Randomly generated FNN · One-layer target FNN | Target-model approximation | Helped |
| Use current LoRA training optimization algorithms | LoRA · Experiments shown in figures | Optimization performance | Hurt |
Surprising findings
- Tuning the final layers of a randomly generated FNN fails to represent even a one-layer target network, regardless of tuning, while LoRA succeeds.
- For FNN experiments, gradient-based LoRA training outperforms the paper's explicit construction at small ranks.
Limitations
- Results concern expressive power only; optimization and generalization are excluded.
- Transformer result is exact construction for same-size target models; approximation bounds for TFNs are left to future work.
- Skip connections are excluded from the Transformer analysis for analytical feasibility.
Experimental setup
Datasets: synthetic random linear and FNN models GLUE benchmark
Benchmarks: GLUE
Metrics: mean squared error approximation error vs LoRA-rank
Baselines: full gradient-based LoRA training final layers tuning
Builds on: LoRA Eckart-Young-Mirsky theorem Expressive power of normalization fine-tuning (Wang et al.)
Closest papers by meaning
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.93)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.93)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.93)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.93)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.93)
- LoRA+: Efficient Low Rank Adaptation of Large Models (ICML 2024 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.