Research map / Architectures and efficiency / Mixture of experts
A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a gate-rescaled Riemannian preconditioning scheme for mixture-of-LoRA-experts fine-tuning, implemented by replacing gate values with their square roots in the forward pass with detached terms, and reports gains over Riemannian SGD/AdamW on QA, GLUE, and VQA.
Problem
MoE-LoRA fine-tuning of foundation models suffers from limited representation and gradient sub-optimality; existing Riemannian preconditioners for LoRA ignore the gate-value scaling of each expert in the mixture.
Approach
Derives that gate values rescale each expert's manifold curvature and squares in the effective gradient projection; corrects this by rescaling per-expert preconditioners with gate values, approximated in practice by forwarding sqrt(g_i) with a detach trick so that gradients scale linearly in g_i.
Key results
- On Llama-3.2-3B QA, gRSGD averages 63.5 vs RSGD 58.5 and gRAdamW averages 76.7 vs RAdamW 75.6.
- On GLUE with Llama-3.2-3B, gRSGD improves RSGD average from 65.16 to 72.90 (about 11.9% relative).
- On LLaVA-v1.5-7B, gRSGD reaches 70% on VMCBench vs 59% for RSGD, while gRAdamW is 75% vs 75% for RAdamW.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add gate-based rescaling to RSGD | MoE-LoRA · Question answering benchmarks | accuracy: 62.8 -> 70.1 (ScienceQA); 52.4 -> 55.4 (CommonsenseQA); 53.2 -> 59.8 (OpenBookQA); 65.7 -> 68.5 (SIQA) | Helped |
| add gate-based rescaling to RAdamW | MoE-LoRA · Question answering benchmarks | accuracy: 82.6 -> 83.8 (ScienceQA); 67.7 -> 68.2 (CommonsenseQA); 70.4 -> 72.4 (OpenBookQA); 81.5 -> 82.3 (SIQA) | Helped |
| add gate-based rescaling to RSGD | MoE-LoRA with Llama-3.2-3B · GLUE average | score: 65.16 -> 72.90 | Helped |
| add gate-based rescaling to RAdamW | MoE-LoRA with Llama-3.2-3B · GLUE average | score: 72.22 -> 74.40 | Helped |
| add gate-based rescaling to RSGD | MoE-LoRA with GLM-4-9B · GLUE average | score: 77.27 -> 80.61 | Helped |
| add gate-based rescaling to RAdamW | MoE-LoRA with GLM-4-9B · GLUE average | score: 77.10 -> 77.60 | Helped |
| add gate-based rescaling to RSGD | MoE-LoRA with LLaVA-v1.5-7B · Visual7W | accuracy: 68% -> 72% | Helped |
| add gate-based rescaling to RSGD | MoE-LoRA with LLaVA-v1.5-7B · VMCBench | accuracy: 59% -> 70% | Helped |
| add gate-based rescaling to RAdamW | MoE-LoRA with LLaVA-v1.5-7B · VMCBench | accuracy: 75% -> 75% | No clear effect |
| add gate-based rescaling to MoELoRA | MoELoRA-SGD · ScienceQA | accuracy: 54.68 -> 70.01 | Helped |
| add gate-based rescaling to MoLA-RSGD | MoLA · ScienceQA | accuracy: 68.03 -> 70.46 | Helped |
| use gate-based forward revision without Riemannian preconditioning (ablation) | MoE-LoRA with conventional SGD · ScienceQA | training loss | No clear effect |
Surprising findings
- Applying the gate-based rescaling to plain SGD without Riemannian preconditioning does not help, indicating the method is coupled to the Riemannian preconditioner.
- Gains are much larger for SGD than for AdamW (AdamW gains are small, around 1-3%).
Limitations
- Gate rescaling is an approximation: the squared gate terms cannot be exactly reproduced per token under batch-averaged loss, so sqrt(g) is used instead, which distorts the forward pass.
- Gains for AdamW are small and the paper notes that improvements are mainly in final optimality rather than speed.
- Experiments use a limited set of models and tasks, with small evaluation sets (100 samples for VMCBench) and no reported seeds or variance.
Experimental setup
Datasets: ScienceQA CommonsenseQA OpenBookQA SIQA GLUE Visual7W VMCBench
Benchmarks: ScienceQA CommonsenseQA OpenBookQA SIQA GLUE Visual7W VMCBench
Metrics: accuracy average accuracy
Baselines: RSGD RAdamW MoELoRA MoLA
Builds on: LoRA Riemannian preconditioners for LoRA MoE-LoRA MoLA AdamW
Closest papers by meaning
- Asymmetry in Low-Rank Adapters of Foundation Models (ICML 2024 · similarity 0.92)
- Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models (ICML 2024 · similarity 0.92)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.90)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.89)
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.89)
- LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.