Research map / Continual learning / Continual fine-tuning of LLMs
Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D-MoLE, which evolves an MLLM's architecture under a parameter budget for continual multimodal instruction tuning by allocating LoRA experts layer-wise via gradient-norm proxies, routing via task autoencoders, and splitting the budget between LLM and vision encoder with a gradient-based curriculum.
Problem
Continual multimodal instruction tuning (CMIT) of MLLMs suffers from catastrophic forgetting under fixed architectures; the paper targets how to allocate new parameters across layers and modalities under a fixed budget as tasks arrive.
Approach
Dynamically allocate LoRA experts to the most gradient-sensitive layers of the LLM and vision encoder (zero-cost proxy), split the parameter budget between modalities by gradient-norm difficulty, and route inputs to previous-task experts using per-task autoencoder reconstruction errors with fallback to the pretrained backbone.
Key results
- D-MoLE reaches 73.87 average and 82.18 last score on the CMIT benchmark, versus 58.79 and 62.04 for the best baseline O-LoRA.
- D-MoLE's BWT is -1.49 against -21.31 for O-LoRA, indicating much less forgetting.
- Reported improvements over the second-best method are 15.08% (AVG), 20.14% (Last), and 19.82% (BWT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use D-MoLE instead of O-LoRA | InternVL2-2B · CMIT benchmark | AVG: 58.79 -> 73.87 | Helped |
| use D-MoLE instead of O-LoRA | InternVL2-2B · CMIT benchmark | BWT: -21.31 -> -1.49 | Helped |
| use D-MoLE instead of O-LoRA | InternVL2-2B · MME-Sum | score: 1646 -> 1754.6 | Helped |
| use D-MoLE instead of O-LoRA | InternVL2-2B · MMMU-Val | score: 32.3 -> 32.7 | Helped |
| use D-MoLE instead of O-LoRA | InternVL2-2B · POPE-Sum | score: 87.6 -> 88.1 | Helped |
| fine-tune only the LLM and freeze the vision encoder (ablation) | D-MoLE · CMIT benchmark | overall performance | Hurt |
| fine-tune only the vision encoder and freeze the LLM (ablation) | D-MoLE · CMIT benchmark | overall performance | Hurt |
| remove gradient-based inter-modal continual curriculum (ablation) | D-MoLE · CMIT benchmark | overall performance | Hurt |
| add gradient-based inter-modal continual curriculum (ablation) | D-MoLE · CMIT benchmark | overall performance | Helped |
| remove dynamic layer-wise expert allocation and allocate LoRA uniformly across layers (ablation) | D-MoLE · CMIT benchmark | overall performance | Hurt |
| use dynamic layer-wise expert allocation instead of uniform LoRA allocation (ablation) | D-MoLE · CMIT benchmark | overall performance | Helped |
| scale the routing thresholds over the tested range (ablation) | D-MoLE · CMIT benchmark, final checkpoint | Avg. Last: 80.37 to 82.18 | No clear effect |
Surprising findings
- Joint-learning, presented as the upper bound, is lower than D-MoLE on average (83.22 vs 73.87 is not satisfied; D-MoLE's Last score 82.18 is close to joint-learning's per-task numbers), suggesting the upper bound is not strict in this setup.
- Increasing O-LoRA's parameter budget (rank 8 or 16) does not meaningfully improve its performance over rank 4.
Limitations
- Task boundaries are assumed known during training; routing relies on dataset-level distribution differences that may blur in harder settings.
- Evaluated only on a single base model (InternVL2-2B) and a nine-dataset benchmark.
Experimental setup
Datasets: VizWiz-VQA IconQA OCR-VQA KVQA PMC-VQA VizWiz-Caption TextCaps Flickr30k SK-VG
Benchmarks: CMIT benchmark (9 datasets) MME MMMU POPE
Metrics: AVG Last BWT top-1 accuracy CIDEr IoU@0.5
Baselines: Seq-FT LwF-LoRA EWC-LoRA Dense MoLE Sparse MoLE MoLA O-LoRA
Builds on: Mixture of LoRA Experts (MoLE) LoRA O-LoRA EWC LwF
Closest papers by meaning
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.91)
- CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.91)
- LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models (CVPR 2025 · similarity 0.90)
- Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning (NeurIPS 2026 · similarity 0.90)
- InduceKV: Fixed-Footprint Continual Adaptation of Multimodal LLMs via Inducing KV Memories (NeurIPS 2026 · similarity 0.89)
- MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.