Research map / Large language models / Instruction and fine-tuning
HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HydraLoRA, a LoRA variant with one shared A matrix and multiple B matrices routed by a trainable MoE router, with clusters of training data (k-means) initializing the B heads, to handle heterogeneous data without domain knowledge.
Problem
LoRA underperforms full fine-tuning on complex, heterogeneous datasets because a single low-rank adapter suffers interference across intrinsic sub-domains or tasks, and prior multi-LoRA methods rely on domain knowledge.
Approach
An asymmetric LoRA with a shared A matrix for commonalities and multiple B matrices for distinct intrinsic components, combined through a learned MoE router; observations show A matrices converge across heads while B matrices diverge, motivating this split.
Key results
- On single-domain tasks with LLaMA2-7B, HydraLoRA (r=8, 0.124% params) reaches 47.22 MMLU, 22.27 GSM8K and 27.43 HumanEval P@10, beating LoRA r=32 (46.59 / 20.67 / 25.89) with half its parameters.
- On the BBH multi-task benchmark, HydraLoRA averages 41.5 (7B) and 44.1 (13B) versus 40.3 and 43.7 for LoRA MoE, using 0.341% parameters versus 2.976%.
- Against LoRA r=32 on GSM8K, HydraLoRA reports 1.96x faster training and 49.6% lower energy cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HydraLoRA instead of LoRA MoE | LLaMA2-7B · BBH | average performance: 40.3 -> 41.5 | Helped |
| Use HydraLoRA instead of LoRA MoE | LLaMA2-13B · BBH | average performance: 43.7 -> 44.1 | Helped |
| Replace LoRA (r=32) with LoRA-Split (4 × 8) (ablation) | LLaMA2-7B · MMLU | score: 46.59 -> 46.94 | Helped |
| Replace LoRA (r=32) with LoRA-Split (4 × 8) (ablation) | LLaMA2-7B · Medical | score: 44.32 -> 45.28 | Helped |
| Replace LoRA (r=32) with LoRA-Split (4 × 8) (ablation) | LLaMA2-7B · Law | score: 40.81 -> 41.35 | Helped |
| Replace LoRA (r=32) with LoRA-Split (4 × 8) (ablation) | LLaMA2-7B · HumanEval | P@1: 17.12 -> 18.20 | Helped |
| Replace LoRA (r=32) with LoRA-Split (4 × 8) (ablation) | LLaMA2-7B · HumanEval | P@10: 25.89 -> 26.85 | Helped |
| Replace LoRA (r=32) with LoRA-Split (4 × 8) (ablation) | LLaMA2-7B · GSM8K | score: 20.67 -> 21.92 | Helped |
| Add the MoE architecture (ablation) | HydraLoRA · language understanding domains | performance | Helped |
| Add the trainable gate function instead of uniform expert weights (ablation) | HydraLoRA · language understanding domains | performance | Helped |
| Use 4 clusters instead of 3 (ablation) | HydraLoRA · MMLU | performance: 0.42% | Helped |
| Use k-means instead of DBSCAN for corpus clustering (ablation) | HydraLoRA · MMLU | performance | No clear effect |
Surprising findings
- Visualizing t-SNE embeddings shows A matrices across heads converge while B matrices differ, suggesting differences come mainly from B.
- Splitting a LoRA into multiple smaller vanilla LoRAs at equal parameter count outperforms a single LoRA of the same total rank.
Limitations
- Training is more expensive than standard PEFT because multiple adapter copies are used; authors report 1-2x more iterations.
- Only LoRA is tested; prompt-tuning and adapter variants are not explored, and pre-training is not evaluated.
- Number of clusters k is chosen via k-means and elbow method, with the sensitivity analysis done only on Dolly-15K and MMLU.
Experimental setup
Datasets: databricks-dolly-15k GenMedGPT ChatDoctor clinic-10k Lawyer-Instruct US-Terms GSM8K CodeAlpaca Flan v2 subset
Benchmarks: MMLU GSM8K HumanEval Big-Bench Hard
Metrics: accuracy pass@1 pass@10 % trainable parameters energy consumption training latency
Baselines: Full fine-tuning Prompt Tuning P-Tuning Prefix Tuning IA3 AdaLoRA LoRA LoraHub LoRA MoE
Builds on: LoRA Mixture-of-Experts LoRAMoE LoraHub AdaLoRA
Closest papers by meaning
- VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks (NeurIPS 2024 · similarity 0.92)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.92)
- BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.91)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.91)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.91)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.