Research map / Vision-language and multimodal / Cross-modal learning
Scaling Omni-modal Pretraining with Multimodal Context: Advancing Universal Representation Learning Across Modalities
Yiyuan Zhang, Handong Li, Jing Liu, Xiangyu Yue
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multimodal context embeddings (Image+Video+Audio pretraining) (ablation) | MiCo · MSRVTT Video-Audio retrieval | R@1: 37.6% -> 42.2% (+4.6) | Improved |
| add multimodal context embeddings (Image+Video+Audio+Depth pretraining) (ablation) | MiCo · MSRVTT Video-Audio retrieval | R@1: 36.3% -> 45.7% (+9.4) | Improved |
| use a ViT for non-text modalities plus an LLM for text instead of modality-specific encoders (ablation) | omni-modal architecture · Flickr retrieval | R@1: 88.4 -> 90.5 | Improved |
| use text-only encoder (BERT) instead of ViT + LLM (ablation) | omni-modal architecture · Flickr retrieval | R@1: 90.5 -> 51.6 | Hurt |
| use LLM decoder only instead of ViT + LLM (ablation) | omni-modal architecture · AudioCaps retrieval | R@1: 49.7 -> 14.7 | Hurt |
| add audio to image pretraining (ablation) | MiCo · AudioCaps retrieval | R@1: 10.2 -> 22.0 | Improved |
| add video and 3D to image+audio pretraining (I+V+A to I+V+A+3D) (ablation) | MiCo · MSRVTT retrieval | R@1: 42.2 -> 45.7 | Improved |
| add audio to image pretraining (ablation) | MiCo · COCO retrieval | R@1: 50.2 -> 46.8 | Hurt |
| scale pretraining data from 1M to 334M pairs (ablation) | MiCo · MSRVTT retrieval | R@1: 44.2 -> 49.1 | Improved |
| add matching loss Llogits to NCE loss (ablation) | MiCo pretraining · MSRVTT retrieval | R@1: 40.1 -> 43.9 | Improved |
| add causal generation loss Lcausal to NCE + logits (ablation) | MiCo pretraining · MSRVTT retrieval | R@1: 43.9 -> 45.7 | Improved |
| scale model from Base-86M to Giant-1.3B (ablation) | MiCo · MSRVTT retrieval | R@1: 45.7 -> 62.5 | Improved |
Closest papers by meaning
- OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.90)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.89)
- Omni-Interactive Universal Embedder (NeurIPS 2026 · similarity 0.89)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.88)
- Scaling Language-centric Omnimodal Representation Learning (NeurIPS 2025 · similarity 0.88)
- XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
