Research map / Architectures and efficiency / Mixture of experts
FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlyLoRA, an implicit rank-wise mixture-of-experts LoRA that uses a frozen sparse random projection (inspired by the fly olfactory circuit) as both router and down-projection, improving single-task accuracy and multi-task merging robustness with fewer activated parameters.
Problem
LoRA suffers from intra-task parameter interference that requires high ranks, and MoE-based LoRA variants add router parameters and fail to prevent inter-task interference when merging LoRA components trained on different tasks.
Approach
Freeze the down-projection A as a sparse random Gaussian projection so its outputs act as an implicit router: top-k rank-1 experts in B are activated by the magnitudes of Ax, removing explicit router parameters; distinct random A_i are nearly orthogonal, which reduces inter-task interference when merging.
Key results
- Single-task on Llama-3.1-8B: FlyLoRA (k=8) reaches 40.88% MMLU vs 38.93% for LoRA r=32, using 0.13% activated parameters vs 1.03%.
- Multi-task weight-averaging merge on Llama-3.1-8B: FlyLoRA MMLU drops only -2.02% vs -4.91% for LoRA r=32 and -4.86% for Split-LoRA.
- Ablation: making A trainable drops multi-task MMLU accuracy from 38.86% to 34.43%, supporting the frozen-A orthogonality argument.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use rank-wise top-k activation with frozen sparse random A (FlyLoRA) instead of LoRA with total rank 32 | Llama-3.1-8B · MMLU | accuracy: 38.93 -> 40.88 | Helped |
| Use rank-wise top-k activation with frozen sparse random A (FlyLoRA) instead of LoRA with total rank 32 | Llama-3.1-8B · ScienceQA | accuracy: 94.01 -> 94.15 | Helped |
| Use rank-wise top-k activation with frozen sparse random A (FlyLoRA) instead of LoRA with total rank 32 | Llama-3.1-8B · GSM8K | accuracy: 56.25 -> 58.76 | Helped |
| Use rank-wise top-k activation with frozen sparse random A (FlyLoRA) instead of Split-LoRA (4×8) | Llama-3.1-8B · HumanEval | Pass@1: 31.28 -> 36.88 | Helped |
| Use FlyLoRA instead of LoRA (r=32) | Qwen-2.5-7B · MMLU | accuracy: 52.07 -> 53.68 | Helped |
| Use FlyLoRA instead of LoRA (r=32) | Qwen-2.5-7B · ScienceQA | accuracy: 95.01 -> 95.55 | Helped |
| Use FlyLoRA instead of LoRA (r=32) | Qwen-2.5-7B · GSM8K | accuracy: 79.23 -> 80.82 | Helped |
| Use FlyLoRA instead of LoRA (r=32) | Qwen-2.5-7B · HumanEval | Pass@1: 52.87 -> 54.34 | Helped |
| Add load balancing (ablation) | FlyLoRA · MMLU | accuracy: 37.56 -> 40.88 | Helped |
| Freeze A rather than make A trainable (ablation) | FlyLoRA · MMLU single-task | accuracy: 40.64 -> 40.88 | No clear effect |
| Freeze A rather than make A trainable (ablation) | FlyLoRA · MMLU after merging | accuracy: 34.43 -> 38.86 | Helped |
| Use FlyLoRA rather than LoRA (r=32) for weight-averaged merging | Llama-3.1-8B · MMLU after merging | accuracy: 34.02 -> 38.86 | Helped |
Surprising findings
- Trainable down-projection A gives nearly the same single-task accuracy as frozen A (40.64 vs 40.88), so the benefit of freezing A appears mainly in multi-task merging.
- On Qwen-2.5-7B, LoRA r=8 and r=32 and Split-LoRA gain accuracy on GSM8K after merging, which the authors attribute to the merge setup rather than a clear loss.
Limitations
- Multi-task evaluation covers only four domains merged by weight averaging and a few merge methods, with mixed results on some merged tasks (e.g., ScienceQA drops of 20-43%).
- Orthogonality and covariance results rely on simplifying assumptions (uniform activation, large n limit) and approximate bounds.
- Experiments are limited to 7-14B models and zero-shot evaluation on a small set of benchmarks.
Experimental setup
Datasets: MMLU auxiliary training set ScienceQA GSM8K CodeAlpaca-20k
Benchmarks: MMLU ScienceQA GSM8K HumanEval
Metrics: accuracy pass@k activated trainable parameter percentage performance drop after merging
Baselines: LoRA (r=8) LoRA (r=32) Split-LoRA (4x8) AdaLoRA SoRA HydraLoRA TIES-MERGING DARE
Builds on: LoRA LoRA-FA Split-LoRA Johnson-Lindenstrauss lemma loss-free load balancing
Closest papers by meaning
- PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning (NeurIPS 2025 · similarity 0.90)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.89)
- SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.88)
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.88)
- Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning (ICLR 2024 · similarity 0.88)
- CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.