Research map / Continual learning / Continual fine-tuning of LLMs
KSS-MoE: Knowledge Space Synergy Framework in Mixture of Experts for Continual Visual Instruction Tuning
Lingyun Song, Ziyao Chen, Kang Pan, Xiaolin Han, Xinbiao Gan, Yudai Pan, Xiaofan Sun, Xiaoqi Wang, Xuequn Shang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use KSS-MoE instead of O-LoRA | LLaVA-1.5-7B continual visual instruction tuning · CoIN (8 tasks) | MAA (%): 52.61 -> 59.99 | Improved |
| use KSS-MoE instead of O-LoRA | LLaVA-1.5-7B continual visual instruction tuning · CoIN (8 tasks) | BWT (%): -19.75 -> -11.25 | Improved |
| add the general expert with orthogonal shared space (ablation) | KSS-MoE · CoIN (8 tasks) | MAA (%): 57.64 -> 59.99 | Improved |
| add the general expert with orthogonal shared space (ablation) | KSS-MoE · CoIN (8 tasks) | BWT (%): -13.54 -> -11.25 | Improved |
| learn the general expert in an orthogonal shared space instead of a vanilla general expert (ablation) | KSS-MoE · CoIN (8 tasks) | MAA (%): 52.61 -> 59.99 | Improved |
| add a vanilla general expert without orthogonal projection (ablation) | KSS-MoE without general expert · CoIN (8 tasks) | MAA (%): 57.64 -> 52.61 | Hurt |
| add expert subspace collaboration instead of naive MoE routing (ablation) | KSS-MoE · CoIN (8 tasks) | MAA (%): 45.00 -> 57.64 | Improved |
| add expert subspace collaboration instead of naive MoE routing (ablation) | KSS-MoE · CoIN (8 tasks) | BWT (%): -21.83 -> -13.54 | Improved |
| increase LoRA rank from 4 to 16 (ablation) | KSS-MoE · CoIN (8 tasks) | MAA (%): 53.42 -> 59.99 | Improved |
| increase LoRA rank from 4 to 8 (ablation) | KSS-MoE · CoIN (8 tasks) | BWT (%): -11.89 -> -15.05 | Hurt |
| increase LoRA rank from 16 to 32 (ablation) | KSS-MoE · CoIN (8 tasks) | MAA (%): 59.99 -> 60.74 | No clear effect |
| vary relevance thresholds epsilon and gamma from 0 to 0.5 (ablation) | KSS-MoE · CoIN (8 tasks) | MAA and BWT | Mixed |
Closest papers by meaning
- DISTMOE: Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning (NeurIPS 2026 · similarity 0.89)
- CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering (CVPR 2025 · similarity 0.89)
- LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.88)
- Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning (ICML 2025 · similarity 0.88)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
