Research map / Vision-language and multimodal / Cross-modal learning
A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Wander, a low-rank sequence multimodal adapter that uses CP-decomposed outer-product fusion to model token-level interactions across any number of modalities in a parameter-efficient way.
Problem
Efficient transfer learning (adapters/prompts) for multimodal models is designed for vision-language pairs, does not extend to more than two modalities, and models only coarse sequence-level interactions while being parameter-heavy.
Approach
Combine outer-product multimodal fusion with CP decomposition applied both to feature and token dimensions, so token-level cross-modal interactions are computed as element-wise products of low-rank projections without forming the high-dimensional tensor; the adapter drops the up-projection and uses a residual with nonlinearity.
Key results
- On UPMC-Food 101, Wander (d=128) reaches 91.8 ACC with 4.9M tunable parameters, matching full fine-tuning (91.8 ACC, 220M).
- On CMU-MOSI, Wander (d=64) reaches 83.2 ACC-2 and 0.71 Corr, outperforming full fine-tuning (82.6 ACC-2, 0.70 Corr) with 0.9M parameters.
- On MSRVTT (seven modalities), Wander (d=16) achieves 57.2 R@5 text-to-video with 0.8M parameters versus 57.2 for full fine-tuning at 134M, with Wander matching full fine-tuning on R@5 and surpassing it on R@10 video-to-text (68.9 vs 68.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Wander (d=128) instead of UniAdapter (d=128) | UPMC-Food 101 transfer learning · UPMC-Food 101 | ACC: 90.8 -> 91.8 | Helped |
| use Wander (d=64) instead of MaPLe | IEMOCAP transfer learning · IEMOCAP | ACC: 73.2 -> 74.7 | Helped |
| use Wander (d=64) instead of MaPLe | CMU-MOSI transfer learning · CMU-MOSI | ACC-2: 82.1 -> 83.2 | Helped |
| use Wander (d=16) instead of MaPLe | MSRVTT retrieval · MSRVTT Text→Video | R@10: 68.2 -> 69.4 | Helped |
| add nonlinearity (ablation) | Wander (d=16) · CMU-MOSI | ACC-2: 81.6 -> 81.8 | Helped |
| add residual connection (ablation) | Wander (d=16) · CMU-MOSI | ACC-2: 81.5 -> 81.8 | Helped |
| add nonlinearity (ablation) | Wander (d=32) · CMU-MOSI | ACC-2: 82.1 -> 82.2 | Helped |
| add residual connection (ablation) | Wander (d=32) · CMU-MOSI | ACC-2: 81.9 -> 82.2 | Helped |
| use sequence fusion instead of vector fusion (ablation) | Wander with down-projection dimension 64 · CMU-MOSI | ACC-2: 80.9 -> 83.2 | Helped |
| increase CP decomposition rank (ablation) | Wander · UPMC-Food 101, CMU-MOSI, IEMOCAP, and MSRVTT | accuracy or R@10 | Helped |
| increase down-projection dimension (ablation) | Wander · IEMOCAP | accuracy: 74.2 -> 74.7 | Helped |
Surprising findings
- Wander matches or exceeds full fine-tuning on CMU-MOSI and MSRVTT despite training under 1M parameters.
- Token-level sequence fusion substantially outperforms vector (CLS-token) fusion on CMU-MOSI (83.2 vs 80.9 ACC-2).
Limitations
- Parameter-count comparisons against unreduced outer-product forms required reducing feature dimensions, so the direct efficiency comparison is on a modified setting.
- Results on CMU-MOSI and IEMOCAP rely on backbones pre-trained on HowTo100M, limiting direct comparability to standard pretrained models.
- Rank choices (R_h, R_t) and d were set empirically, with gains described only qualitatively.
Experimental setup
Datasets: UPMC-Food 101 CMU-MOSI IEMOCAP MSRVTT HowTo100M CMU-MOSEI
Benchmarks: UPMC-Food 101 CMU-MOSI IEMOCAP MSRVTT
Metrics: accuracy F1 ACC-7 MAE Pearson correlation R@5 R@10 MdR MnR
Baselines: LoRA Adapter P-tuning MaPLe PMF UniAdapter Aurora Full fine-tuning
Builds on: adapter tuning CP decomposition low-rank multimodal fusion LoRA MMT
Closest papers by meaning
- UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling (ICLR 2024 · similarity 0.88)
- Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.88)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.87)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.87)
- MMA: Multi-Modal Adapter for Vision-Language Models (CVPR 2024 · similarity 0.87)
- MokA: Multimodal Low-Rank Adaptation for MLLMs (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.