Research map / Large language models / Instruction and fine-tuning
Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GOAT, an SVD-structured LoRA mixture-of-experts whose experts are initialized from distinct singular-value segments, with a theoretically derived scaling factor that aligns its optimization with full fine-tuned upcycled MoE.
Problem
LoRA underperforms full fine-tuning, and combining SVD-based initialization with LoRA MoE raises weight misalignment and complex gradient dynamics that prior single-LoRA methods do not face.
Approach
Initialize each LoRA MoE expert with a different SVD singular-value segment so the router adaptively selects pre-trained priors, subtract a residual weight so the equivalent weight matches the pre-trained weight, and scale the low-rank factors (sqrt(1/(s·rho))) so the equivalent gradient matches full fine-tuning.
Key results
- On seven image classification datasets with CLIP ViT-B/32 at total rank 8, GOAT reaches 81.49 average accuracy versus 82.25 for Full FT and 77.09 for LoRA.
- On commonsense reasoning with LLaMA2-7B, GOAT averages 82.73, exceeding the best single-LoRA baseline KaSA (81.53) and HydraLoRA (81.12).
- On GSM8K with LLaMA2-7B, GOAT scores 60.20 versus 59.36 for Full FT and 56.63 for MoLoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GOAT's adaptive SVD-segment initialization and MoE scaling instead of MoLoRA | CLIP ViT-B/32 fine-tuning · StanfordCars | accuracy: 50.83 -> 53.50 | Helped |
| Use GOAT instead of DoRA | Llama-2-7B fine-tuning · MT-Bench | score: 5.97 -> 6.01 | Helped |
| Use GOAT instead of HydraLoRA | Llama-2-7B fine-tuning · GSM8K | accuracy: 57.39 -> 60.20 | Helped |
| Use GOAT instead of MoLoRA | Llama-2-7B fine-tuning · HumanEval | score: 24.83 -> 25.61 | Helped |
| Use GOAT instead of KaSA | LLaMA2-7B fine-tuning · Commonsense reasoning average | accuracy: 81.53 -> 82.73 | Helped |
| Use GOAT instead of MoLoRA | RoBERTa-large fine-tuning · GLUE average | score: 89.52 -> 89.76 | Helped |
| Use adaptive selected SVD segments instead of principal singular-value initialization (ablation) | GOAT · Image classification average | accuracy: 81.11 -> 81.49 | Helped |
| Use adaptive selected SVD segments instead of minor singular-value initialization (ablation) | GOAT · Image classification average | accuracy: 81.14 -> 81.49 | Helped |
| Use adaptive selected SVD segments instead of randomly selected SVD segments (ablation) | GOAT · Image classification average | accuracy: 81.22 -> 81.49 | Helped |
| Add SVD initialization (ablation) | GOAT MoE · Image classification average | accuracy: 81.06 -> 81.49 | Helped |
| Add MoE architecture instead of using a single LoRA (ablation) | LoRA with GOAT SVD fragments · Image classification average | accuracy: 77.62 -> 80.35 | Helped |
| Add MoE scaling (ablation) | GOAT · Image classification average | accuracy: 80.35 -> 81.49 | Helped |
Surprising findings
- Several SVD-based single-LoRA methods (PiSSA, MiLoRA) underperform plain LoRA on some benchmarks, e.g., PiSSA gets 85.00 average on GLUE versus 88.46 for LoRA.
- Zero (non-SVD) initialization of the LoRA MoE scores 81.06 versus 81.49 with SVD segments, a smaller gain than the MoE scaling effect suggests.
Limitations
- Theoretical scaling derivation relies on approximations (SGD, small updates, first-step gradient) and is extended to general settings heuristically.
- Full FT MoE is only evaluated on image classification and GLUE due to memory requirements.
- Sensitive to balance-loss coefficient; too-low coefficients cause expert imbalance and degraded performance.
Experimental setup
Datasets: Commonsense170K WizardLM MetaMathQA Code-Feedback GLUE StanfordCars DTD EuroSAT GTSRB RESISC45 SUN397 SVHN
Benchmarks: GLUE MT-Bench GSM8K HumanEval BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA
Metrics: accuracy MT-Bench score pass@1
Baselines: LoRA DoRA PiSSA MiLoRA rsLoRA LoRA-Dash NEAT KaSA MoLoRA AdaMoLE HydraLoRA Full FT MoE
Builds on: LoRA PiSSA MiLoRA KaSA rsLoRA
Closest papers by meaning
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.93)
- GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.92)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.92)
- AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping (NeurIPS 2025 · similarity 0.92)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.92)
- Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.