Research map / Continual learning / Continual fine-tuning of LLMs

Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning

NeurIPS 2026 · Continual learning › Continual fine-tuning of LLMs · approach: Mixture-of-experts for multimodal continual tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DRAPE, which generates instance-specific soft prompts from instruction and visual patch features for multimodal continual instruction tuning, combined with null-space gradient projection on the shared projector and CLIP prototype routing for task-label-free generator selection.

Problem

Multimodal continual instruction tuning of MLLMs must acquire new tasks while limiting catastrophic forgetting; existing module-composition methods route task-level prompts or LoRA experts and ignore within-task sample variation.

Approach

Replaces fixed task-level prompt pools with a per-task generator that synthesizes query-image conditioned soft prompts via instruction-derived queries cross-attending to visual patch features, plus null-space gradient projection on the shared projector and CLIP prototype routing.

Key results

• On CoIN with LLaVA-v1.5-7B, DRAPE reaches 67.48 average accuracy vs 59.09 for the strongest baseline ProgLoRA.
• On UCIT, DRAPE reaches 69.41 average accuracy vs 66.54 for SEFE.
• Ablation on CoIN: removing null-space projection drops average to 64.86 (-2.62); removing cross-modal attention drops to 66.31 (-1.17).

Surprising findings

• Oracle routing (68.17) is only slightly above the learned prototype router (67.48), while no-routing drops to 57.65.
• Prompt-based DRAPE shows near-zero or negative backward transfer on UCIT (average B=0.04), indicating almost no forgetting.

Limitations

• Only validated in the prompt-tuning setting; not extended to LoRA-style adaptation.
• Routing errors occur between related tasks (GQA vs VQAv2, ScienceQA to OCR-VQA/VQAv2).

Experimental setup

Datasets: CoIN UCIT

Benchmarks: CoIN UCIT

Metrics: average accuracy backward transfer mean accuracy

Baselines: Finetune CODA-Prompt DualPrompt L2P MoELoRA Continual LLaVA ModalPrompt ProgLoRA LoRA-FT O-LoRA CL-MoE HiDe

Builds on: L2P DualPrompt CODA-Prompt ModalPrompt ProgLoRA

Closest papers by meaning

• Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.91)
• Convolutional Prompting meets Language Models for Continual Learning (CVPR 2024 · similarity 0.90)
• Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning (ICML 2025 · similarity 0.90)
• CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.89)
• InduceKV: Fixed-Footprint Continual Adaptation of Multimodal LLMs via Inducing KV Memories (NeurIPS 2026 · similarity 0.89)
• Exploring the Transferability of Visual Prompting for Multimodal Large Language Models (CVPR 2024 · similarity 0.89)