Research map / Architectures and efficiency / Mixture of experts
MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
Multi-task adaptation of large foundation models suffers from task conflict (interference between new tasks) and task oblivion (catastrophic forgetting of original capabilities), which existing LoRA-based MoE adapters do not address well.
Approach
Reinterprets the SVD of a pretrained weight matrix as a static MoE of orthogonal rank-one experts (u_d v_d^T), makes the expert weights input- and task-dependent via a hybrid task/sample router that adjusts singular values, and adds a shared Householder orthogonal adapter; orthogonality of experts limits cross-task interference and preserving the column space of W limits forgetting.
Key results
- On CSR-MTL with LLaMA-3.1 8B, MoORE (L=8) reaches 85.11% overall vs. 84.34% for LoRAMoE and 79.54% for LoRA, with 2.84% of parameters.
- On NLU-MTL, MoORE (L=0) reaches 89.08% overall vs. 88.08% for LoRAMoE.
- On OR-MTL, MoORE shows an average drop of only 1.31% relative to the base LLaMA-3.1 8B, versus 4.26% for LoRAMoE and 47.09% for LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MoORE instead of LoRAMoE | LLaMA-3.1 8B · CSR-MTL | Overall (%): 84.34 -> 85.11 | Helped |
| use MoORE instead of LoRAMoE | LLaMA-3.1 8B · NLU-MTL | Overall (%): 88.08 -> 89.08 | Helped |
| use MoORE instead of LoRAMoE | Qwen-2.5 7B · CSR-MTL | Overall (%): 86.17 -> 86.63 | Helped |
| use MoORE instead of LoRAMoE | LLaMA-3.1 8B · OR-MTL | Overall (%): 57.76 -> 60.71 | Helped |
| add the orthogonal adapter (increase L from 0 to 8) (ablation) | MoORE on LLaMA-3.1 8B · CSR-MTL | Overall (%): 84.98 -> 85.11 | Helped |
| add the orthogonal adapter (increase L from 0 to 2) (ablation) | MoORE on LLaMA-3.1 8B · NLU-MTL | Overall (%): 89.08 -> 89.04 | No clear effect |
| increase Householder reflections from L=0 to L=16 (ablation) | MoORE on Qwen-2.5 7B · CSR-MTL | Overall (%): 86.31 -> 86.63 | Helped |
| use task-level routing only instead of full MoORE (ablation) | MoORE on LLaMA-3.1 8B · CSR-MTL | Overall (%): 79.55; 81.80; 83.06; 83.53 vs 85.11 | Hurt |
| use sample-level routing only instead of full MoORE (ablation) | MoORE on LLaMA-3.1 8B · CSR-MTL | Overall (%): 82.30; 82.91; 83.57; 83.80 vs 85.11 | Hurt |
| use the orthogonal adapter only instead of full MoORE (ablation) | MoORE on LLaMA-3.1 8B · CSR-MTL | Overall (%): 80.76; 82.15; 82.98; 83.90 vs 85.11 | Hurt |
| add the full combination of task-level routing, sample-level routing, and orthogonal adapter (ablation) | MoORE on LLaMA-3.1 8B · CSR-MTL | Overall (%): 79.55; 83.80; 83.90 -> 85.11 | Helped |
| adapt on CSR-MTL before evaluating instead of evaluating the base model | MoORE · OR-MTL HumanEval | score: 9.75% | Helped |
Surprising findings
- Applying the orthogonal adapter did not improve performance on NLU-MTL, suggesting it helps mainly on knowledge-heavy commonsense tasks.
- MoORE improved HumanEval by 9.75 points over the base model after adaptation on commonsense tasks, which the authors attribute to possible task relevance.
Limitations
- MoORE uses many experts (D rank-one experts) and learns dense expert weights, unlike sparse classic MoE, making further efficiency reduction challenging.
- The theoretical trade-off between task number, conflict and oblivion is not quantified and remains open.
- No error bars or repeated runs due to limited compute.
Experimental setup
Datasets: CSR-MTL NLU-MTL OR-MTL
Benchmarks: ARC-C ARC-E BoolQ OBQA PIQA SIQA HellaSwag WinoGrande CommonsenseQA CoLA MNLI MRPC
Metrics: accuracy Matthews correlation performance drop relative to base model
Baselines: LoRA LoRAMoE MixLoRA MoSLD MTL-LoRA HydraLoRA Full fine-tuning
Builds on: LoRA LoRAMoE HMoRA Householder reflection adaptation (HRA) O-LoRA
Closest papers by meaning
- FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts (NeurIPS 2025 · similarity 0.84)
- Parameter-Efficient Multi-Task Model Fusion with Partial Linearization (ICLR 2024 · similarity 0.83)
- Distribution-Aligned Decoding for Efficient LLM Task Adaptation (NeurIPS 2025 · similarity 0.83)
- MTLoRA: Low-Rank Adaptation Approach for Efficient Multi-Task Learning (CVPR 2024 · similarity 0.83)
- Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning (ICLR 2024 · similarity 0.83)
- Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.