Research map / Architectures and efficiency / Pruning and efficient tuning
DoRA: Weight-Decomposed Low-Rank Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DoRA, a PEFT method that decomposes pretrained weights into magnitude and direction, applying LoRA only to the directional component, to narrow the accuracy gap between LoRA and full fine-tuning without inference overhead.
Problem
LoRA and its variants leave an accuracy gap relative to full fine-tuning, and prior work attributes this mainly to the limited number of trainable parameters without analyzing the underlying learning differences.
Approach
Use a weight decomposition analysis (magnitude vs. direction, inspired by Weight Normalization) to show LoRA's update pattern differs from full fine-tuning; then decompose W into a trainable magnitude vector and a directional matrix updated via LoRA, so the method mirrors FT's learning pattern while remaining mergeable at inference.
Key results
- On commonsense reasoning, DoRA achieves 78.4% average on LLaMA-7B vs. LoRA 74.7%, and 85.2% vs. 80.8% on LLaMA3-8B.
- On VL-BART image/video-text tasks, DoRA averages 77.4 on image-text (matching FT 77.3 vs. LoRA 76.5) and 85.4 on video-text vs. LoRA 83.5.
- On LLaVA-1.5-7B visual instruction tuning, DoRA averages 67.6 vs. LoRA 66.9 and FT 66.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add magnitude-direction weight decomposition with LoRA directional updates | LoRA on LLaMA-7B · Commonsense reasoning (8-task average) | accuracy: 74.7 -> 78.4 | Helped |
| add magnitude-direction weight decomposition with LoRA directional updates | LoRA on LLaMA-13B · Commonsense reasoning (8-task average) | accuracy: 80.5 -> 81.5 | Helped |
| add magnitude-direction weight decomposition with LoRA directional updates | LoRA on LLaMA2-7B · Commonsense reasoning (8-task average) | accuracy: 77.6 -> 79.7 | Helped |
| add magnitude-direction weight decomposition with LoRA directional updates | LoRA on LLaMA3-8B · Commonsense reasoning (8-task average) | accuracy: 80.8 -> 85.2 | Helped |
| add magnitude-direction weight decomposition with LoRA directional updates | LoRA on VL-BART · Image-text understanding (4-task average) | Avg.: 76.5 -> 77.4 | Helped |
| add magnitude-direction weight decomposition with LoRA directional updates | LoRA on VL-BART · Video-text understanding (4-task average) | Avg.: 83.5 -> 85.4 | Helped |
| add magnitude-direction weight decomposition with LoRA directional updates | LoRA on LLaVA-1.5-7B · Visual instruction tuning (7-task average) | Avg.: 66.9 -> 67.6 | Helped |
| use VeRA for the directional update | VeRA on LLaMA-7B · MT-Bench | Score: 4.3 -> 5.0 | Helped |
| use VeRA for the directional update | VeRA on LLaMA2-7B · MT-Bench | Score: 5.5 -> 6.0 | Helped |
| halve the LoRA rank in DoRA (ablation) | DoRA on LLaMA-7B · Commonsense reasoning (8-task average) | accuracy: 74.7 -> 77.5 | Helped |
| vary rank to 4 (ablation) | DoRA versus LoRA on LLaMA-7B · Commonsense reasoning (8-task average) | Avg.: 39.5 -> 61.9 | Helped |
| vary rank to 8 (ablation) | DoRA versus LoRA on LLaMA-7B · Commonsense reasoning (8-task average) | Avg.: 40.7 -> 77.9 | Helped |
Surprising findings
- Full fine-tuning on LLaVA-1.5-7B underperforms LoRA on average, suggesting FT may overfit.
- DoRA's learned magnitude/direction updates show a negative magnitude-direction correlation, unlike LoRA's positive correlation.
Limitations
- Gains over LoRA are smaller when FT itself underperforms LoRA (e.g., LLaVA-1.5-7B).
- Experiments mostly limited to LLaMA-family models and specific multimodal setups; some results are quoted from prior papers.
- Analysis of learning patterns is based on a single case study (VL-BART) and correlation statistics.
Experimental setup
Datasets: BoolQ PIQA SIQA HellaSwag WinoGrande ARC OBQA VQAv2 GQA NLVR2 MSCOCO TVQA
Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA VQAv2 GQA NLVR2 COCO Caption
Metrics: accuracy CIDEr MT-Bench score exact match GPU memory
Baselines: LoRA full fine-tuning Prefix tuning Series adapter Parallel adapter VeRA ChatGPT
Builds on: LoRA Weight Normalization VeRA QLoRA
Closest papers by meaning
- Parameter-Efficient Fine-Tuning with Discrete Fourier Transform (ICML 2024 · similarity 0.89)
- SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.89)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.89)
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.88)
- AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.