Research map / Vision-language and multimodal / Cross-modal learning
LLMs Can Evolve Continually on Modality for $\mathbb{X}$-Modal Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PathWeave, which extends a pretrained vision LLM (X-InstructBLIP) to new modalities (video, audio, depth, 3D) via continual learning on uni-modal data using an Adapter-in-Adapter design with MoE gating, evaluated on a new MCL benchmark.
Problem
Expanding multimodal LLMs to new modalities normally requires joint-modal pretraining and tuning, which is computationally expensive and needs re-access to historical data; the paper addresses continual modality expansion without joint-modal data while avoiding catastrophic forgetting.
Approach
An Adapter-in-Adapter (AnA) scheme: frozen uni-modal LoRA-style adapters per modality, plus cross-modal adapters formed by inserting small in-adapters into earlier frozen uni-modal adapters, combined with an MoE-style gating that weights the paths per modality.
Key results
- Ours achieves zero forgetting (F=0.00) across all modality transitions, while Continual-FT forgets substantially (e.g., 25.50 on image after video, 68.19 after depth).
- Average transfer score across in-domain datasets is 63.74, versus 33.91 for the best prior CL method EProj (+29.83).
- Parameter training is reduced by at least 98.73% versus X-InstructBLIP and OneLLM (0.8-2.4M trainable parameters vs 189.91M+ and 7B+).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PathWeave instead of EProj for in-domain modality expansion | continual learning on modalities · in-domain datasets, average T-hat | average T-hat: +29.83 | Helped |
| use PathWeave instead of Continual-FT | continual learning on modalities · in-domain historical modalities | average F-hat: -3.00 | Helped |
| add MoE-based gating (ablation) | PathWeave · in-domain transfer-learning datasets | average score | Helped |
| add In-Adapter (ablation) | PathWeave · in-domain transfer-learning datasets | average score | Helped |
| add MoE-based gating and In-Adapter to Continual-Adapter (ablation) | continual modality adapters · in-domain transfer-learning datasets | average score: +2.64 | Helped |
| add MoE-based gating (ablation) | continual modality adapters · audio and depth transfer (AudioCaps Val) | score: 52.1 -> 64.0; 92.5 -> 64.0 | Mixed |
| add In-Adapter (ablation) | continual modality adapters · in-domain transfer-learning datasets | average score: 55.15 -> 57.79 | Helped |
| add In-Adapter (ablation) | continual modality adapters · AudioCaps QA | score: 27.7 -> 46.5 | Helped |
| add In-Adapter (ablation) | continual modality adapters · AudioCaps Val | score: 71.4 -> 64.0 | Hurt |
| use PathWeave instead of Continual-FT | continual learning on modalities · out-of-domain datasets | average T-hat: +18.95 | Mixed |
Surprising findings
- On the Image-to-Video step, EProj (47.60) and full continual fine-tuning transfer ability slightly better than the proposed method on that transition (45.08 vs 47.60 for EProj).
Limitations
- Only five modalities (image, video, audio, depth, 3D) were explored; not all real-world modalities are covered.
- Implicit interaction between modalities does not support cross-modal joint reasoning tasks in an incremental manner.
- Comparisons with MLLMs use unified settings and only the instruction tuning stage, so results are not directly comparable to published numbers.
Experimental setup
Datasets: MS COCO CapFilt14M CC12M SBU Captions Visual Genome AOK-VQA OK-VQA OCR-VQA VQAv2 LLaVA-150K MSRVTT MSRVTT-QA
Benchmarks: MCL benchmark (COCO, MSRVTT, MSVD, AudioCaps, ESC50, ClothoAQA, CC3M, LLaVA-50K, NYU v2, SUN RGB-D, ModelNet, Cap3D) GQA
Metrics: T_m (transfer score on new modality) F_m (forgetting rate) CIDEr/accuracy on per-dataset scores training time (s/it) GPU memory (G) trainable parameters
Baselines: Continual-FT WiSE-FT L2 Reg&WE EProj X-InstructBLIP OneLLM X-LLM ChatBridge
Builds on: X-InstructBLIP LoRA Q-Former EProj
Closest papers by meaning
- First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training (NeurIPS 2025 · similarity 0.89)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.88)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.88)
- Evaluating Cross-Modal Reasoning Ability and Problem Charactaristics with Multimodal Item Response Theory (ICLR 2026 · similarity 0.88)
- HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding (ICML 2025 · similarity 0.88)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.