Research map / Architectures and efficiency / Pruning and efficient tuning
Decoupling Angles and Strength in Low-rank Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeLoRA, a LoRA variant that normalizes each rank-1 component of the low-rank update and scales it by a learnable boundary, decoupling angular learning from adaptation strength for robust PEFT.
Problem
LoRA is sensitive to learning rate and degrades under extended training, while bounded methods like ETHER are limited to very low rank and fixed transformation strength.
Approach
Normalize the inner r-dimensional bottleneck of the low-rank update (B Xi A with unit-norm rank-1 components), bound its Frobenius norm via a learnable scale lambda times the pretrained weight norm divided by r, so the update is learning-rate-independent in magnitude and its strength is set by lambda.
Key results
- On GLUE with RoBERTa-base, DeLoRA (0.3M params) reaches 85.6 average, matching full finetuning (85.6) and above LoRA (84.7).
- On instruction tuning with LLaMA-2-7B, DeLoRA r=8 averages 41.42 vs DoRA 40.73 and LoRA 40.19, best on three of four tasks.
- In subject-driven generation, DeLoRA r=16 with tuned hyperparameters achieves DINO 0.693 and CLIP-I 0.820, comparable to LoRA and DoRA; LoRA and DoRA degrade at 4x base learning rate while DeLoRA stays robust.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add weights-norm scaling to normalized LoRA updates (ablation) | LoRA-derived DeLoRA · Subject-driven image generation ablation | DINO: 0.682 -> 0.701 | Helped |
| Add weights-norm scaling to normalized LoRA updates (ablation) | LoRA-derived DeLoRA · Subject-driven image generation ablation | CLIP-I: 0.809 -> 0.825 | Helped |
| Add a controllable boundary to ETHER+ (ablation) | ETHER+ one-sided · Subject-driven image generation ablation | DINO: 0.624 -> 0.678 | Helped |
| Add a controllable boundary to ETHER+ (ablation) | ETHER+ one-sided · Subject-driven image generation ablation | CLIP-I: 0.746 -> 0.810 | Helped |
| Add normalization with a controllable boundary (ablation) | LoRA · Subject-driven image generation ablation | DINO: 0.674 -> 0.682 | Helped |
| Add normalization with a controllable boundary (ablation) | LoRA · Semantic Map to Image ablation | Acc.: 64.95 -> 65.82 | Helped |
| Increase ETHER+ update rank (ablation) | ETHER+ · Semantic Map to Image ablation | Acc. | Helped |
| Use DeLoRA instead of LoRA | RoBERTa-base · GLUE | Avg: 84.7 -> 85.6 | Helped |
| Use DeLoRA instead of DoRA | Stable Diffusion · Subject-driven image generation | CLIP-I: 0.819 -> 0.820 | Helped |
| Use DeLoRA instead of DoRA | LLaMA-2-7B · Instruction tuning | Avg: 40.73 -> 41.42 | Helped |
| Fix the DoRA magnitude term (ablation) | DoRA r=16 · Subject-driven image generation | DINO: 0.683 -> 0.681 | Hurt |
| Fix the DoRA magnitude term (ablation) | DoRA r=16 · Subject-driven image generation | CLIP-I: 0.820 -> 0.822 | Helped |
Surprising findings
- DoRA with a fixed magnitude term does not reproduce DeLoRA's robustness and can diverge more at high learning rates.
- DeLoRA's weights reach the boundary and stop diverging during prolonged training, whereas LoRA and DoRA overshoot.
Limitations
- Layer-wise weight norm scaling is a heuristic; the authors note more sophisticated layer-wise treatment is future work.
- Learnable boundary theoretically allows unbounded Frobenius norm; divergence avoidance is empirical rather than guaranteed.
- Ablations and robustness analyses are on small-scale settings (3 subjects, 10% of data) and a single model family for image generation.
Experimental setup
Datasets: DreamBooth subject dataset ADE20K GLUE Alpaca MMLU ARC TruthfulQA
Benchmarks: DreamBooth subject-driven generation (DINO, CLIP-I) ADE20K semantic map to image (mIoU, Acc, FID) GLUE MMLU ARC TruthfulQA
Metrics: DINO CLIP-I mIoU accuracy FID Matthews correlation average accuracy
Baselines: LoRA DoRA ETHER ETHER+ OFT BitFit IA3 LoReFT RED Adapter DreamBooth
Builds on: LoRA ETHER DoRA DreamBooth PiSSA
Closest papers by meaning
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.91)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.90)
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.90)
- Parameter-Efficient Fine-Tuning with Discrete Fourier Transform (ICML 2024 · similarity 0.89)
- Asymmetry in Low-Rank Adapters of Foundation Models (ICML 2024 · similarity 0.89)
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.