Research map / Vision-language and multimodal / Cross-modal learning
OmniVec2 - A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniVec2, a transformer network with modality-specific tokenizers, a shared transformer backbone, cross-attention fusion, and a three-stage training scheme (unimodal masked pretraining, pairwise multimodal masked pretraining, pairwise multitask supervised training) evaluated on 25 datasets across 12+ modalities.
Problem
Training a single network that handles many modalities and tasks jointly, with knowledge sharing across modalities, without per-modality encoders or unified task heads.
Approach
Modality-specific tokenizers feed a shared BERT-style transformer bottleneck; pairs of modalities are fused via cross-attention during pretraining and multitask training, and each modality gets its own ViT-Tiny task heads. Training samples modality pairs and one task per modality rather than fully joint training.
Key results
- Achieves 94.6% top-1 on iNaturalist-2018 and 65.1% on Places-365, above OmniVec (93.8%, 63.5%).
- Reports 93.6% top-1 on Kinetics-400 and 53.1% on Moments in Time, outperforming listed baselines.
- Ablation shows modality-specific tokenizers and task-specific heads each improve over unified designs; multimodal pretraining beats unimodal pretraining by about 16% on iNaturalist and K400.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multimodal multitask pretraining instead of unimodal pretraining (ablation) | OmniVec2 · iNaturalist-2018 | top-1 accuracy: 74.2 -> 90.3 | Helped |
| use multimodal multitask pretraining instead of unimodal pretraining (ablation) | OmniVec2 · Kinetics-400 | top-1 accuracy: 78.6 -> 88.4 | Helped |
| use multimodal multitask pretraining instead of unimodal pretraining (ablation) | OmniVec2 · ESC50 | top-1 accuracy: 82.4 -> 92.4 | Helped |
| use modality-specific rather than unified tokenizers (ablation) | OmniVec2 with unified task heads · iNaturalist-2018, Kinetics-400, ESC50 | top-1 accuracy: 80.1, 81.8, 82.7 -> 85.4, 84.8, 86.8 | Helped |
| use modality-specific rather than unified tokenizers (ablation) | OmniVec2 with task-specific heads · iNaturalist-2018, Kinetics-400, ESC50 | top-1 accuracy: 86.1, 85.2, 87.0 -> 90.3, 88.4, 92.4 | Helped |
| use task-specific rather than unified task heads (ablation) | OmniVec2 with modality-specific tokenizers · iNaturalist-2018, Kinetics-400, ESC50 | top-1 accuracy: 85.4, 84.8, 86.8 -> 90.3, 88.4, 92.4 | Helped |
| use OmniVec2 rather than OmniVec | Image and video classification benchmarks · iNaturalist-2018, Places-365, Kinetics-400, Moments in Time | top-1 accuracy: 93.8 -> 94.6; 63.5 -> 65.1; 91.1 -> 93.6; 49.8 -> 53.1 | Helped |
| use OmniVec2 rather than OmniVec | OmniVec2 · PCQM4M-LSC val | MAE: 0.1234 -> 0.1397 | Hurt |
Surprising findings
- On Graph (PCQM4M-LSC), the method is far better than Meta-Transformer (0.1397 vs 0.8863 val MAE) but still behind the specialized Graphormer (0.1234).
- Zero-shot-style results on YouCook2 and MSR-VTT are near fully fine-tuned OmniVec and SM numbers, with the method reported as using significantly less data than SM.
Limitations
- Many comparisons use different protocols; unseen-dataset results freeze the base embeddings and use simpler heads, so they are not directly comparable to full fine-tuning baselines.
- Graph and tabular results are close to but not above specialized state of the art; the method was not designed for graph data.
- Text results on GLUE are mixed, with second-best on only three of five tasks.
Experimental setup
Datasets: AudioSet Something-Something v2 English Wikipedia ImageNet1K SUN RGB-D ModelNet40 iNaturalist-2018 Places-365 Kinetics-400 Moments in Time ESC50 S3DIS
Benchmarks: iNaturalist-2018 Places-365 Kinetics-400 Moments in Time ESC50 ModelNet40-C S3DIS DialogueSUM GLUE UCF-101 HMDB51 Oxford-IIIT Pets
Metrics: top-1 accuracy mIoU error rate ROUGE Recall@10 MAE MSE R@1 mAP overall accuracy
Baselines: OmniVec Omnivore MetaFormer Meta-Transformer Omni-MAE VATT UniformerV2 InternVideo BEATs AST Swin3D Graphormer
Builds on: Uni-Perceiver OmniVec MetaFormer BERT data2vec
Closest papers by meaning
- Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities (CVPR 2024 · similarity 0.86)
- FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning (NeurIPS 2026 · similarity 0.85)
- ViT-Lens: Towards Omni-modal Representations (CVPR 2024 · similarity 0.85)
- 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities (NeurIPS 2024 · similarity 0.85)
- Omni-Interactive Universal Embedder (NeurIPS 2026 · similarity 0.84)
- OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.