Research map / Architectures and efficiency / Pruning and efficient tuning
Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoTo, a progressive schedule that linearly increases the per-adapter activation probability of LoRA adapters over the first 75% of training, improving single-task accuracy, multi-task merging, pruning robustness and training speed across LoRA variants.
Problem
LoRA adapters often converge to suboptimal, layer-imbalanced minima near initialization, which hurts generalization and downstream operations such as adapter merging and pruning.
Approach
Stochastically deactivate each layer's LoRA adapter with a Bernoulli probability that grows linearly from 0 to 1 over the first 75% of training, then keep all adapters active; the authors justify this via a Jensen bound on a binomially weighted subnetwork loss and a Shapley-value analysis of adapter contributions.
Key results
- On commonsense reasoning with LLaMA-3-8B at rank 32, LoRA-CoTo averages 85.02% vs 80.79% for vanilla LoRA, and HiRA-CoTo reaches 87.00% vs 86.72% for HiRA.
- On LLaMA-2-7B multi-task merging, linear weight fusion of CoTo adapters improves average accuracy from 47.17% to 58.53%.
- Wall-clock training time drops by 7.20% for LoRA, 23.69% for DoRA and 24.21% for HiRA on a single A6000 GPU for math reasoning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add progressive adapter activation to LoRA | LoRA on ViT-B/16 · 11 image classification tasks (Avg) | accuracy (%): 82.95 -> 83.48 | Helped |
| Add progressive adapter activation to DoRA | DoRA on ViT-B/16 · 11 image classification tasks (Avg) | accuracy (%): 83.45 -> 83.93 | Helped |
| Add progressive adapter activation to HiRA | HiRA on ViT-B/16 · 11 image classification tasks (Avg) | accuracy (%): 83.98 -> 84.34 | Helped |
| Add progressive adapter activation to LoRA | LoRA on LLaMA-2-7B · Commonsense170K tasks (Avg) | accuracy (%): 77.61 -> 80.63 | Helped |
| Add progressive adapter activation to DoRA | DoRA on LLaMA-2-7B · Commonsense170K tasks (Avg) | accuracy (%): 79.69 -> 80.64 | Helped |
| Add progressive adapter activation to HiRA | HiRA on LLaMA-3-8B · Commonsense170K tasks (Avg) | accuracy (%): 86.72 -> 87.00 | Helped |
| Add progressive adapter activation to LoRA-Pro | LoRA-Pro on LLaMA-2-7B · GSM8K | accuracy (%): 54.23 -> 57.16 | Helped |
| Add progressive adapter activation | Independent LoRA solutions · Commonsense interpolation at λ=0.5 | accuracy (%): 39% -> 79% | Helped |
| Train task adapters with progressive activation before linear weight fusion | LLaMA-2-7B task-specific adapters · 9 generative language task merging (Avg) | accuracy (%): 47.17 -> 58.53 | Helped |
| Train task adapters with progressive activation before LoRA-LEGO merging | LLaMA-2-7B task-specific adapters · 9 generative language task merging (Avg) | accuracy (%): 62.21 -> 67.19 | Helped |
| Train task adapters with progressive activation before linear weight fusion | LLaMA-2-13B task-specific adapters · 9 generative language task merging (Avg) | accuracy (%): 49.86 -> 66.84 | Helped |
| Train task adapters with progressive activation before linear fusion | DeBERTa-v3 task-specific adapters · 6 discriminative language task merging (Avg, LR 5e-4) | accuracy (%): 59.35 -> 63.41 | Helped |
Surprising findings
- Vanilla LoRA's linear-interpolation accuracy at lambda=0.5 drops to 39% on commonsense tasks while LoRA-CoTo maintains 79%.
- Adapter contribution analysis shows vanilla LoRA concentrates 69% of Shapley contribution in the highest 4 of 12 layers.
Limitations
- Gains are reported on a fixed 75% first-phase schedule and linear ramp; the authors only compare against exponential and sine alternatives on DTD.
- Theory is a Jensen-type bound on the expected loss and does not prove the claimed LMC or flatness benefits directly.
- Some results are mixed; e.g., on DTD and in some merging settings (Ensemble on LLaMA-2-7B) gains are small or negative.
Experimental setup
Datasets: Commonsense170K MetaMathQA GSM8K 11 image classification datasets (CLIP-LoRA setup) GLUE Flan Collection DreamBooth DTD
Benchmarks: 11 image classification datasets ARC-c ARC-e BoolQ HellaSwag OBQA PIQA SIQA WinoGrande GSM8K CoLA MNLI
Metrics: accuracy linear interpolation accuracy pruning accuracy wall-clock training time Shapley value
Baselines: LoRA DoRA HiRA PiSSA rsLoRA LoRA+ LoRA-Pro LoRA-LEGO ZipLoRA
Builds on: LoRA DoRA HiRA LoRA-Pro Stochastic Depth
Closest papers by meaning
- LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.88)
- Flora: Low-Rank Adapters Are Secretly Gradient Compressors (ICML 2024 · similarity 0.88)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.88)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.88)
- Asymmetry in Low-Rank Adapters of Foundation Models (ICML 2024 · similarity 0.88)
- Sparse High Rank Adapters (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.