Research map / Continual learning / Continual fine-tuning of LLMs
CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CL-MoE, a LoRA-based mixture-of-experts framework on LLaVA-7B with dual (task- and instance-level) routers and momentum expert updates to mitigate catastrophic forgetting in continual VQA on VQA v2 split into 10 question-type tasks.
Problem
Continual visual question answering with MLLMs suffers from catastrophic forgetting when trained sequentially on non-stationary VQA task streams, and existing continual methods built on small models (VL-T5) lack commonsense knowledge and still forget.
Approach
Combines a Dual-Router MoE (instance-level and task-level routers selecting local and global experts) with a Dynamic Momentum MoE that updates expert parameters via momentum between previous and newly tuned weights, with task-shared vs task-specific expert distinction, plus a cluster-based task-index inference.
Key results
- CL-MoE on LLaVA-7B reaches 51.34% AP and -0.02% AF on VQA v2 10-task continual setting, versus 43.49 AP / 9.10 AF for VQACL and 36.98 AP / 20.69 AF for vanilla LLaVA.
- Ablation: MMoE alone gives 48.43 AP; RMoE alone 41.23 AP; full model 51.34 AP.
- Reverse task order yields 57.08 AP and -1.44 AF for CL-MoE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CL-MoE instead of VQACL | LLaVA-7B · VQA v2 (10 tasks) | AP: 43.49 -> 51.34 | Helped |
| apply CL-MoE instead of VQACL | LLaVA-7B · VQA v2 (10 tasks) | AF: 9.10 -> -0.02 | Helped |
| add MMoE to the Vanilla model (ablation) | LLaVA-7B · VQA v2 (10 tasks) | AP: 32.51 -> 48.43 | Helped |
| add MMoE to the Vanilla model (ablation) | LLaVA-7B · VQA v2 (10 tasks) | AF: 20.69 -> 3.02 | Helped |
| add RMoE to the Vanilla model (ablation) | LLaVA-7B · VQA v2 (10 tasks) | AP: 32.51 -> 41.23 | Helped |
| add RMoE to the Vanilla model (ablation) | LLaVA-7B · VQA v2 (10 tasks) | AF: 20.69 -> 11.09 | Helped |
| use both MMoE and RMoE instead of MMoE alone (ablation) | LLaVA-7B · VQA v2 (10 tasks) | AP: 48.43 -> 51.34 | Helped |
| use both MMoE and RMoE instead of RMoE alone (ablation) | LLaVA-7B · VQA v2 (10 tasks) | AP: 41.23 -> 51.34 | Helped |
| use 4 experts instead of 8 (ablation) | CL-MoE · VQA v2 (10 tasks) | AP: 51.34 -> 50.19 | Hurt |
| select top 1 expert instead of top 2 (ablation) | CL-MoE · VQA v2 (10 tasks) | AP: 51.34 -> 50.64 | Hurt |
| select top 3 experts instead of top 2 (ablation) | CL-MoE · VQA v2 (10 tasks) | AP: 51.34 -> 51.22 | No clear effect |
| reverse the task order (ablation) | CL-MoE · VQA v2 (10 tasks) | AP: 51.34 -> 57.08 | Helped |
Surprising findings
- Using RMoE alone without MMoE performs poorly, as knowledge is lost during training without momentum updates.
- Rehearsal-based methods show worse forgetting on LLaVA than on VL-T5, while regularization methods show the opposite trend.
Limitations
- Gap to the Multitask upper bound remains substantial (51.34 vs 55.15 type numbers on VQA v2 AP).
- Evaluated only on VQA v2 with question-type task splits; no other benchmarks or task domains.
- Hyperparameters (gamma, beta, n, K) tuned on the same benchmark reported.
Experimental setup
Datasets: VQA v2
Benchmarks: VQA v2
Metrics: accuracy Average Performance (AP) Average Forgetting (AF)
Baselines: Vanilla EWC MAS ER DER VS VQACL Multitask VL-T5
Builds on: LoRA LLaVA VQACL Mixture of Experts
Closest papers by meaning
- Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.90)
- AVQACL: A Novel Benchmark for Audio-Visual Question Answering Continual Learning (CVPR 2025 · similarity 0.89)
- Cross-modal Information Flow in Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.88)
- Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (NeurIPS 2024 · similarity 0.88)
- CLOVA: A Closed-LOop Visual Assistant with Tool Usage and Update (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.