Research map / Continual learning / Continual fine-tuning of LLMs
CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CoIN, a benchmark of 10 datasets over 8 tasks for continual instruction tuning of MLLMs, with Truth Alignment and LLM-graded Reasoning Capability metrics, and proposes MoELoRA to reduce forgetting.
Problem
Continual instruction tuning of multimodal LLMs has been overlooked and lacks public benchmarks; it is unclear whether MLLMs forget reasoning knowledge or instruction-following ability when tuned sequentially on diverse tasks.
Approach
A sequential instruction-tuning benchmark built from public vision-language datasets (adding classification and grounding tasks), paired with two evaluation axes (exact-match Truth Alignment and LLM-graded Reasoning Capability) that separate instruction-following forgetting from knowledge forgetting, plus a mixture-of-LoRA-experts method (MoELoRA) to mitigate forgetting.
Key results
- LLaVA sequential fine-tuning reaches BWT of -32.62 on Truth Alignment, while reasoning capability drops only -10.88 BWT, supporting the claim that instruction-following rather than knowledge is forgotten.
- MoELoRA with 8 experts on LLaVA improves MAA from 32.97 (single LoRA) to 42.76 and BWT from -32.62 to -25.91 under the random order.
- MiniGPT-v2 shows near-zero forgetting in Truth Alignment (BWT +6.04) but underfits on some tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoELoRA instead of the baseline sequential fine-tuning | LLaVA on CoIN with 40% data · CoIN Truth Alignment | MAA: 33.18 -> 41.05 | Helped |
| Use MoELoRA instead of the baseline sequential fine-tuning | LLaVA on CoIN with 40% data · CoIN Truth Alignment | BWT: -24.85 -> -22.50 | Helped |
| Increase the number of MoELoRA experts from 1 to 8 (ablation) | LLaVA · CoIN Truth Alignment | MAA: 32.97 -> 42.76 | Helped |
| Use alphabetical instead of random task order (ablation) | LLaVA · CoIN Truth Alignment | MAA: 32.97 -> 31.08 | Hurt |
| Use diverse task-specific templates instead of original templates (ablation) | LLaVA · CoIN Truth Alignment | MAA: 32.97 -> 32.92 | No clear effect |
| Randomly choose among 10 instruction templates instead of original templates (ablation) | LLaVA · CoIN Truth Alignment | MAA: 32.97 -> 38.37 | Helped |
| Increase training data volume from 10% to 40% (ablation) | LLaVA · CoIN Truth Alignment | MAA: 30.27 -> 33.18 | Helped |
| Increase training data volume from 40% to 80% (ablation) | LLaVA · CoIN Truth Alignment | MAA: 33.18 -> 30.00 | Hurt |
| Increase the backbone from 7B to 13B (ablation) | LLaVA · CoIN Truth Alignment | MAA: 32.97 -> 39.43 | Helped |
| Increase LoRA rank from 128 to 192 (ablation) | LLaVA with 40% data · CoIN Truth Alignment | MAA: 33.18 -> 38.31 | Helped |
| Increase LoRA rank from 128 to 256 (ablation) | LLaVA with 40% data · CoIN Truth Alignment | BWT: -24.85 -> -19.8 | Helped |
| Sequentially fine-tune rather than multi-task fine-tune | LLaVA on CoIN · CoIN Truth Alignment | BWT: -32.62 | Hurt |
Surprising findings
- MiniGPT-v2 retains Truth Alignment with positive BWT while LLaVA and Qwen-VL forget heavily.
- Randomly sampling from 10 instruction templates improves retention substantially (BWT -31.75 vs -32.62 gain in MAA from 32.97 to 38.37), whereas simply diversifying templates per task has minimal effect.
Limitations
- Only 7-9B MLLMs studied; no full fine-tuning and no larger models in main experiments.
- Most MLLMs are LLaMA-based, limiting architectural diversity.
- Task diversity is limited mainly to image QA plus classification and grounding.
Experimental setup
Datasets: VQAv2 VizWiz ScienceQA TextVQA GQA OCR-VQA ImageNet RefCOCO RefCOCO+ RefCOCOg
Benchmarks: CoIN
Metrics: accuracy IoU>0.5 accuracy for grounding MAA BWT LLM-graded reasoning score (0-10)
Baselines: Multi-task Zero-shot LwF EWC Qwen-VL MiniGPT-v2
Builds on: LoRA MoELoRA LwF EWC
Closest papers by meaning
- Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning (ICML 2025 · similarity 0.91)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.91)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.90)
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.90)
- LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models (CVPR 2025 · similarity 0.90)
- Large Continual Instruction Assistant (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.