Research map / Vision-language and multimodal / CLIP and zero-shot
Mixture of LoRA Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoLE, which treats each layer of multiple trained LoRAs as an expert and learns per-layer gating weights (with a balancing loss) to compose them for NLP and vision-language generation.
Problem
Composing multiple trained LoRAs either degrades the base model's generative ability (direct linear sums) or loses individual LoRA characteristics (normalized weights), while reference tuning-based composition requires costly retraining and manual masks.
Approach
Treat each layer of each trained LoRA as an expert and learn a gating function per layer over LoRA outputs, keeping all LoRAs and the base model frozen, with a gating balancing loss to prevent gate collapse; masking LoRAs at inference re-normalizes the remaining weights.
Key results
- On Big-Bench Hard, MoLE averages 38.3 EM across 8/24/48/128 LoRAs versus 35.9 for LoRAHub, the best baseline.
- In V&L multi-concept generation with 3 concepts, MoLE reaches 0.759 text-alignment versus 0.728 for SVDiff and 0.678 for normalized linear arithmetic composition.
- Generalization from NLI-trained LoRAs to BBH yields 40.9 average EM for MoLE versus 38.5 for LoRAHub.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compose LoRA experts with learned gating rather than normalized linear arithmetic composition | Stable Diffusion V2.1 · V&L multi-concept generation (3 concepts, average) | Text-alignment: 0.678 -> 0.752 | Helped |
| compose LoRA experts with learned gating rather than SVDiff | Stable Diffusion V2.1 · V&L multi-concept generation (3 concepts, average) | Image-alignment: 0.715 -> 0.743 | Helped |
| compose LoRA experts with learned gating rather than Custom | Stable Diffusion V2.1 · V&L multi-concept generation (3 concepts, average) | Image-alignment: 0.760 -> 0.743 | Hurt |
| use MoLE instead of LoRAHub | Flan-T5 · BBH (average EM, 48 LoRAs) | EM: 36.9 -> 39.4 | Helped |
| add gating balancing loss (ablation) | MoLE on Flan-T5 · ANLI-R1, ANLI-R2, ANLI-R3, QNLI, WNLI (average) | Average score: 77.57 -> 78.07 | Helped |
| replace gating balancing loss with increased temperature τ1 (ablation) | MoLE on Flan-T5 · ANLI-R1, ANLI-R2, ANLI-R3, QNLI, WNLI (average) | Average score: 78.07 -> 77.45 | Hurt |
| replace gating balancing loss with increased temperature τ2 (ablation) | MoLE on Flan-T5 · ANLI-R1, ANLI-R2, ANLI-R3, QNLI, WNLI (average) | Average score: 78.07 -> 76.71 | Hurt |
| replace gating balancing loss with increased temperature τ3 (ablation) | MoLE on Flan-T5 · ANLI-R1, ANLI-R2, ANLI-R3, QNLI, WNLI (average) | Average score: 78.07 -> 76.35 | Hurt |
| use block-wise instead of matrix-wise gating (ablation) | MoLE on Stable Diffusion V2.1 · V&L multi-concept generation | Text-alignment: 0.731 -> 0.766 | Helped |
| use block-wise instead of network-wise gating (ablation) | MoLE on Stable Diffusion V2.1 · V&L multi-concept generation | Text-alignment: 0.722 -> 0.766 | Helped |
| use block-wise instead of layer-wise gating (ablation) | MoLE on Stable Diffusion V2.1 · V&L multi-concept generation | Image-alignment, Concept 1: 0.727 -> 0.726 | No clear effect |
Surprising findings
- Gating functions tend to collapse toward one early well-performing LoRA (e.g., 68% weight on one of three), which motivates the balancing loss.
- LoRAHub's optimizer zeroes out many LoRA weights when the number of LoRAs grows, underutilizing them.
Limitations
- Performance of all composition methods, including MoLE, declines when composing very many LoRAs (e.g., 128).
- Reference tuning-based composition is not directly compared because it relies on position masks.
- Hardware and compute details are not reported.
Experimental setup
Datasets: FLAN datasets ANLI-R1 ANLI-R2 ANLI-R3 QNLI WNLI Big-Bench Hard
Benchmarks: Big-Bench Hard ANLI QNLI WNLI CLIP text-image alignment (multi-concept generation)
Metrics: exact match CLIP text-alignment CLIP image-alignment
Baselines: normalized linear arithmetic composition SVDiff LoRAHub PEMs Custom Diffusion Textual Inversion
Builds on: LoRA Mixture-of-Experts DreamBooth LoRAHub SVDiff
Closest papers by meaning
- SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.85)
- Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.85)
- IterIS: Iterative Inference-Solving Alignment for LoRA Merging (CVPR 2025 · similarity 0.84)
- Cached Multi-Lora Composition for Multi-Concept Image Generation (ICLR 2025 · similarity 0.84)
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.84)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.