AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Scaling Omni-modal Pretraining with Multimodal Context: Advancing Universal Representation Learning Across Modalities

Yiyuan Zhang, Handong Li, Jing Liu, Xiangyu Yue

ICCV 2025 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add multimodal context embeddings (Image+Video+Audio pretraining) (ablation)MiCo · MSRVTT Video-Audio retrievalR@1: 37.6% -> 42.2% (+4.6)Improved
add multimodal context embeddings (Image+Video+Audio+Depth pretraining) (ablation)MiCo · MSRVTT Video-Audio retrievalR@1: 36.3% -> 45.7% (+9.4)Improved
use a ViT for non-text modalities plus an LLM for text instead of modality-specific encoders (ablation)omni-modal architecture · Flickr retrievalR@1: 88.4 -> 90.5Improved
use text-only encoder (BERT) instead of ViT + LLM (ablation)omni-modal architecture · Flickr retrievalR@1: 90.5 -> 51.6Hurt
use LLM decoder only instead of ViT + LLM (ablation)omni-modal architecture · AudioCaps retrievalR@1: 49.7 -> 14.7Hurt
add audio to image pretraining (ablation)MiCo · AudioCaps retrievalR@1: 10.2 -> 22.0Improved
add video and 3D to image+audio pretraining (I+V+A to I+V+A+3D) (ablation)MiCo · MSRVTT retrievalR@1: 42.2 -> 45.7Improved
add audio to image pretraining (ablation)MiCo · COCO retrievalR@1: 50.2 -> 46.8Hurt
scale pretraining data from 1M to 334M pairs (ablation)MiCo · MSRVTT retrievalR@1: 44.2 -> 49.1Improved
add matching loss Llogits to NCE loss (ablation)MiCo pretraining · MSRVTT retrievalR@1: 40.1 -> 43.9Improved
add causal generation loss Lcausal to NCE + logits (ablation)MiCo pretraining · MSRVTT retrievalR@1: 43.9 -> 45.7Improved
scale model from Base-86M to Giant-1.3B (ablation)MiCo · MSRVTT retrievalR@1: 45.7 -> 62.5Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy