Research map / Large language models / Instruction and fine-tuning
MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MeteoRA, which embeds 28 existing task-specific LoRA adapters into a single base LLM (LLaMA2-13B, LLaMA3-8B) via a full-mode MoE with trainable gating networks and faster MoE forward kernels, enabling autonomous adapter switching including on composite sequential tasks.
Problem
Multiple LoRA adapters embedded in one LLM normally require explicit task selection, which hinders autonomous task sensing and timely adapter switching, especially for composite tasks with several sub-problems.
Approach
Attach a trainable gating network to every linear layer that routes each token to top-k frozen existing LoRA adapters acting as MoE experts, trained with an autoregressive loss plus a LoRA-classification loss, and accelerate the many-adapter forward pass with bmm and a fused Triton kernel.
Key results
- On LLaMA3-8B, MeteoRA (T1-1k) reaches average accuracy 0.811 vs 0.817 for the PEFT reference and 0.767 for the single trained LoRA-F baseline.
- On LLaMA2-13B, MeteoRA (T1-1k) averages 0.755 accuracy vs 0.762 for PEFT and 0.730 for LoRA-F; merge baselines (Avg Merge, TIES, DARE, LoraHub) fall to 0.31-0.39.
- The bmm-torch kernel gives roughly 4x speedup over the loop-original implementation, though it remains about 2.5x slower than single-lora; the loop approach can be up to 10x slower.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MeteoRA top-1 routing with 1k gate-training samples per task instead of PEFT's explicit adapter activation | LLaMA2-13B · 28 tasks (average) | accuracy: 0.762 -> 0.754 | No clear effect |
| Use MeteoRA top-1 routing with 1k gate-training samples per task instead of PEFT's explicit adapter activation | LLaMA3-8B · 28 tasks (average) | accuracy: 0.817 -> 0.811 | No clear effect |
| Use MeteoRA top-2 routing instead of top-1 routing (ablation) | MeteoRA · 28 tasks (average), LLaMA2-13B | accuracy: 0.755 -> 0.748 | Mixed |
| Reduce gate-training samples from 1k to 100 per task and use top-2 routing (ablation) | MeteoRA · 28 tasks (average), LLaMA2-13B | accuracy: 0.748 -> 0.758 | Mixed |
| Reduce gate-training samples from 1k to 5 per task and use top-2 routing (ablation) | MeteoRA · 28 tasks (average), LLaMA2-13B | accuracy: 0.748 -> 0.740 | Hurt |
| Reduce gate-training samples from 1k to 100 per task and use top-2 routing (ablation) | MeteoRA · 28 tasks (average), LLaMA3-8B | accuracy: 0.806 -> 0.783 | Hurt |
| Reduce gate-training samples from 1k to 5 per task and use top-2 routing (ablation) | MeteoRA · 28 tasks (average), LLaMA3-8B | accuracy: 0.806 -> 0.773 | Hurt |
| Use MeteoRA top-2 routing instead of LoRA-B | LLaMA3-8B · composite-10 | Avg Correct: 2.95 -> 3.75 | Helped |
| Use MeteoRA top-2 routing instead of LoRA-B | LLaMA3-8B · composite-3 | Avg Correct: 1.31 -> 1.49 | Helped |
| Replace loop-original forward with bmm-torch or bmm-triton acceleration (ablation) | MeteoRA forward pass · 28 tasks | speedup: ~4 × | Helped |
| Replace bmm-torch with the custom Triton kernel (ablation) | MeteoRA forward pass · 28 tasks | time efficiency: 80\% | Hurt |
| Directly replace updated LoRA adapters without retraining MeteoRA gates (ablation) | MeteoRA · test set | performance | No clear effect |
Surprising findings
- The Triton kernel is reported to require extra handling because the Triton block dimension constraint (>=16) conflicts with vector-matrix products at rank 8.
- Knowledge-fusion test (GSM8K translated to Italian) showed no gain over the GSM8K LoRA alone, which the authors attribute to the base model already handling Italian.
Limitations
- Gating networks must be retrained when any LoRA adapter is updated, due to distribution shift in hidden states.
- Inference overhead remains substantially above single-LoRA even with the proposed acceleration kernels.
- Knowledge-fusion across domains showed no improvement over the single relevant adapter.
Experimental setup
Datasets: BigBench (22 tasks) News-Commentary GSM8K CNN/DailyMail Alpaca
Benchmarks: BigBench subtasks News-Commentary GSM8K CNN/DailyMail Alpaca composite-3 composite-5 composite-10
Metrics: accuracy BLEU ROUGE-1 ROUGE-2 ROUGE-L
Baselines: PEFT LoRA-F LoRA-B Avg Merge TIES DARE Arrow LoraHub
Builds on: LoRA Mixtral PEFT S-LoRA LoraHub
Closest papers by meaning
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.90)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.90)
- BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.90)
- SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.90)
- Efficient Multi-task LLM Quantization and Serving for Multiple LoRA Adapters (NeurIPS 2024 · similarity 0.89)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.