Research map / Vision-language and multimodal / Cross-modal learning
Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multimodal Pathway, which improves a ViT on one modality (image, video, point cloud, audio) using irrelevant unimodal data from another modality via Cross-Modal Re-parameterization that merges auxiliary weights into the target model at no inference cost.
Problem
Whether a transformer for one modality can be improved using completely irrelevant (unpaired, unlabeled) data from another modality, unlike prior multimodal methods that require paired or interleaved data.
Approach
Build pathways between a target transformer and an auxiliary transformer pretrained on another modality by adding each target linear layer's weight to its auxiliary counterpart scaled by a learnable cross-modal scale initialized at zero; after training, the weights are merged so inference cost is unchanged.
Key results
- M2PT-Point raises ImageNet fine-tuned accuracy from 83.3 to 83.9 over MAE-pretrained ViT-B (+0.7%) and COCO box AP from 47.3 to 50.0 (+5.7%).
- Under from-scratch training, M2PT-Point improves ImageNet accuracy from 76.5 to 81.9 (+7.1%).
- M2PT-Video reaches 82.3 top-1 on Kinetics-400, above VideoMAE's 81.5, and M2PT-Point reaches 35.6 on AudioSet-2k, above AudioMAE's 35.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an irrelevant point-cloud auxiliary transformer | ImageNet-1K recognition · ImageNet-1K | top-1 accuracy: 83.3 -> 83.9 | Helped |
| Add an irrelevant point-cloud auxiliary transformer | Image recognition transfer · MS COCO val | APbox: 47.3 -> 50.0 | Helped |
| Add an irrelevant video auxiliary transformer | Point-cloud segmentation · ShapeNetPart | class mIoU: 84.2 -> 85.6 | Helped |
| Add an irrelevant point-cloud auxiliary transformer | AudioSet recognition · AudioSet-2k | top-1 accuracy: 35.3 -> 35.6 | Helped |
| Add an irrelevant point-cloud auxiliary transformer | Video recognition · Kinetics-400 | top-1 accuracy: 81.5 -> 82.3 | Helped |
| Apply Cross-Modal Re-parameterization to attention QKV only (ablation) | M2PT on ImageNet-1K · ImageNet-1K | top-1 accuracy: 83.3 -> 83.4 | Helped |
| Apply Cross-Modal Re-parameterization to attention projection only (ablation) | M2PT on ImageNet-1K · ImageNet-1K | top-1 accuracy: 83.3 -> 83.6 | Helped |
| Apply Cross-Modal Re-parameterization to the first FFN layer only (ablation) | M2PT on ImageNet-1K · ImageNet-1K | top-1 accuracy: 83.3 -> 83.6 | Helped |
| Apply Cross-Modal Re-parameterization to the second FFN layer only (ablation) | M2PT on ImageNet-1K · ImageNet-1K | top-1 accuracy: 83.3 -> 83.7 | Helped |
| Apply Cross-Modal Re-parameterization to all transformer linear layers (ablation) | M2PT on ImageNet-1K · ImageNet-1K | top-1 accuracy: 83.3 -> 83.9 | Helped |
| Initialize Cross-Modal Scale to 10−2 and freeze it (ablation) | M2PT on ImageNet-1K · ImageNet-1K | top-1 accuracy: 83.9 -> 83.5 | Hurt |
| Initialize Cross-Modal Scale to 10−2 instead of 0 (ablation) | M2PT on ImageNet-1K · ImageNet-1K | top-1 accuracy: 83.9 -> 83.6 | Hurt |
Surprising findings
- Pure point cloud data (ShapeNet) improves image, video, and audio recognition despite having no sample-level relation to those modalities.
- Using a point cloud auxiliary model with frozen target linear weights (only scales and classifier trainable) yields a larger relative gain (+3.4% on ImageNet) than full fine-tuning.
Limitations
- Theoretical explanation for why irrelevant cross-modal data helps is not provided.
- Training cost roughly doubles in trainable parameters and increases training time relative to the baseline.
- Only transformer architectures are explored; extension to CNNs and cross-architecture pathways is left for future work.
Experimental setup
Datasets: ImageNet-1K MS COCO 2017 ADE20K ShapeNetPart ShapeNet PartNet AudioSet-2M AudioSet-2k Kinetics-400 Kinetics-700
Benchmarks: ImageNet-1K MS COCO 2017 ADE20K ShapeNetPart PartNet AudioSet-2k Kinetics-400
Metrics: top-1 accuracy box AP mask AP mIoU class mIoU instance mIoU
Baselines: MAE SemMAE MFF Point-BERT Point-MAE AudioMAE AST SSAST VideoMAE MViTv2 TimeSFormer SlowFast
Builds on: Vision Transformer MAE Structural Re-parameterization AudioMAE Point-MAE VideoMAE
Closest papers by meaning
- OmniVec2 - A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning (CVPR 2024 · similarity 0.86)
- Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion (ICML 2024 · similarity 0.85)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.85)
- A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter (AAAI 2025 · similarity 0.85)
- Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.85)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.