Research map / Continual learning / Continual fine-tuning of LLMs

Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models

NeurIPS 2025 · Continual learning › Continual fine-tuning of LLMs · approach: Mixture-of-experts for multimodal continual tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies inter-modal forgetting in continual instruction tuning of unified multimodal generative models and proposes MoDE, which decouples text (T-MoE LoRA) and image (V-Adapter LoRA) updates with logit-level knowledge distillation on the image adapter.

Problem

Continual instruction tuning of unified multimodal generative models (UMGMs) suffers catastrophic forgetting both within the text modality (intra-modal, understanding tasks) and across modalities (inter-modal, image generation degraded by understanding-task tuning).

Approach

Decouple modality-specific updates with separate LoRA parameter sets (a sparse MoE-LoRA for text tokens and a single LoRA for image tokens) so text-loss gradients do not touch the image adapter, and anchor the image adapter to the frozen pre-trained model via KD on image-token logits.

Key results

• On Chameleon with five VQA/classification tasks, MoDE reaches 33.47% ACC and 25.99% forgetting, versus 32.86% ACC and 30.95% forgetting for CL-MoE, while achieving FID 53.74 close to zero-shot 52.13.
• DualPrompt has the lowest forgetting (6.82%) but only 31.92% ACC, since it learns the hard tasks poorly.
• With Janus-Pro-1B on six tasks, MoDE attains 53.02% ACC and 14.48% forgetting, versus 43.18% and 21.96% for CL-MoE.

Measured effects reported in the paper

Surprising findings

• Adding knowledge distillation to coupled baselines (MoELoRA, CL-MoE) reduces visual generation forgetting but hurts understanding accuracy, showing KD alone is insufficient without modality decoupling.
• DualPrompt's low forgetting figure is misleading because it barely learns new tasks (ImageNet best accuracy 24.55%).

Limitations

• Chameleon original checkpoints are unavailable, so a community reimplementation is used.
• Evaluation covers only a small set of tasks and sequences, with image generation measured by CLIP-based scores and FID on CustomConcept101.
• Key theoretical claim relies on a first-order orthogonality argument that holds only by construction of disjoint parameter blocks, and the reported cosine analysis is for the specific setup.

Experimental setup

Datasets: ScienceQA TextVQA GQA VizWiz ImageNet LAION-5B RefCOCO RefCOCO+ RefCOCOg CustomConcept101

Benchmarks: ScienceQA TextVQA GQA VizWiz ImageNet CustomConcept101 RefCOCO RefCOCO+ RefCOCOg

Metrics: accuracy average forgetting CLIP image alignment CLIP text alignment FID

Baselines: Seq LoRA Model Tailor DualPrompt MoELoRA CL-MoE Joint Training

Builds on: LoRA MoELoRA CL-MoE Model Tailor DualPrompt

Closest papers by meaning

• Quantifying Cross-Modality Memorization in Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.88)
• A Closer Look at Multimodal Representation Collapse (ICML 2025 · similarity 0.87)
• Wings: Learning Multimodal LLMs without Text-only Forgetting (NeurIPS 2024 · similarity 0.87)
• Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Process (NeurIPS 2026 · similarity 0.87)
• Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning (NeurIPS 2026 · similarity 0.87)