Research map / Large language models / Language, safety and interpretability
Towards Modular LLMs by Building and Reusing a Library of LoRAs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds LoRA libraries from Flan-v2 tasks via MBC (clustering tasks by LoRA weight similarity) and reuses them zero-shot via Arrow routing (per-token, per-layer routing by SVD-derived prototypes), evaluated on Phi-2 and Mistral-7B.
Problem
How to build a library of parameter-efficient adapters (LoRAs) from multi-task data and reuse it for zero-shot and supervised generalization to new tasks without jointly training the router and experts.
Approach
MBC clusters tasks using cosine similarity of privately trained LoRA weights and trains one adapter per cluster; Arrow routes each token and layer to LoRA experts using the top right singular vector of each expert's A_i B_i^T as a prototype, requiring no training or training data.
Key results
- On Phi-2 zero-shot, MBC-Arrow reaches 67.4% average accuracy over 10 held-out tasks, vs 63.8% Base and 65.5% FullFT, and 1.2 points above the strongest baseline MHR-mu (65.9%).
- Private-Arrow on Phi-2 reaches 66.6%, surpassing FullFT (65.5%) with independently trained experts and post-hoc routing.
- On Mistral-7B, Arrow routing improves the Private library by 2.5 points over uniform routing (64.4% vs 61.9%); on 12 SNI supervised adaptation (100% data), MBC-Poly reaches 78.9 Rouge-L on Phi-2 vs 76.9 for Private-mu.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Cluster tasks using LoRA parameter similarity rather than random example grouping (ablation) | Phi-2 MBC library · 10 held-out tasks | accuracy: 64.8 -> 67.1 | Helped |
| Cluster whole tasks rather than randomly partitioning task examples (ablation) | Phi-2 library · 10 held-out tasks | accuracy: 64.8 -> 66.4 | Helped |
| Use task representation embeddings instead of LoRA parameter similarity for clustering (ablation) | Phi-2 library · 10 held-out tasks | accuracy: 66.1 -> 67.1 | Hurt |
| Use Arrow rather than uniform routing | Phi-2 Private library · 10 held-out tasks | accuracy: 64.8 -> 66.6 | Helped |
| Use Arrow rather than uniform routing | Phi-2 MBC library · 10 held-out tasks | accuracy: 67.1 -> 67.4 | Helped |
| Use Arrow rather than uniform routing | Mistral 7B Private library · 10 held-out tasks | accuracy: 2.5% | Helped |
| Use Arrow rather than uniform routing | Mistral 7B MBC library · 10 held-out tasks | accuracy: 64.6 -> 63.9 | Hurt |
| Use MBC clustering rather than private per-task adapters | Phi-2 library with uniform routing · 10 held-out tasks | accuracy: 64.8 -> 67.1 | Helped |
| Use MBC clustering rather than private per-task adapters | Mistral 7B library with uniform routing · 10 held-out tasks | accuracy: 61.9 -> 64.6 | Helped |
| Use MBC-Poly instead of randomly grouped tasks with Poly routing | Phi-2 · 12 SNI tasks, 100% data | Rouge-L: 76.7 -> 78.9 | Helped |
| Use MBC-Poly rather than MBC-μ, enabling supervised routing | Phi-2 · 12 SNI tasks, 10% data | Rouge-L: 67.0 -> 68.2 | Helped |
| Tune expert parameters as well as routing coefficients (ablation) | MBC-Poly adaptation · 12 SNI tasks, 0.5% data | Rouge-L: 44.8 -> 50.2 | Helped |
Surprising findings
- Uniform averaging (mu routing) of adapters is competitive, and for MBC libraries simple averaging outperforms more sophisticated routing such as Arrow and TP.
- Arrow routing gives no gain for the MBC library on Mistral and for the SNI 12-task out-of-distribution evaluation on MBC, where mu routing performed better.
Limitations
- Focuses only on LoRA adapters on attention layers; extension to other adapter types left to future work.
- Experiments limited to small models (Phi-2 2.8B, Mistral 7B) and a subsampled 10k-examples-per-task Flan-v2 setting.
- Gains of Arrow are diminishing for small libraries and the linearity of LoRA experts may explain weaker benefits than sparse MoE routing.
Experimental setup
Datasets: Flan-v2 Super-NaturalInstructions (SNI) WinoGrande HellaSwag PIQA BoolQ OpenbookQA ARC-easy ARC-challenge HumanEval MBPP BBH
Benchmarks: PIQA BoolQ WinoGrande HellaSwag ARC-easy ARC-challenge HumanEval OpenbookQA BBH MBPP SNI (12 held-out tasks)
Metrics: accuracy Rouge-L
Baselines: Base FullFT Shared Poly MHR Private-mu LoraHub pi-tuning RandTask RandExamples Embeddings
Builds on: LoRA Polytropon Multi-Head Routing LoraHub Flan-v2
Closest papers by meaning
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.89)
- Universal Model Routing for Efficient LLM Inference (ICLR 2026 · similarity 0.89)
- Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.88)
- GraphRouter: A Graph-based Router for LLM Selections (ICLR 2025 · similarity 0.88)
- RouteLLM: Learning to Route LLMs from Preference Data (ICLR 2025 · similarity 0.87)
- Efficient Multi-task LLM Quantization and Serving for Multiple LoRA Adapters (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.