Research map / Vision-language and multimodal / Cross-modal learning
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LLM-integrated omni-modality captions with simple concatenation (add LLM) (ablation) | VAST-27M omc pretraining · MSRVTT retrieval | Recall@1: 44.4 -> 53.0 | Helped |
| add omni-modality pretraining on VAST-27M vs no pretraining (ablation) | VAST fine-tuned with V+A+S · MSRVTT retrieval | Recall@1: 37.6 -> 53.0 | Helped |
| add audio and subtitle tracks to vision-only (no pretraining) (ablation) | VAST baseline · YouCook2 retrieval | Recall@1: 6.2 -> 32.6 | Helped |
| add subtitle track to vision-only (no pretraining) (ablation) | VAST baseline · VALOR-32K retrieval | Recall@1: 29.1 -> 29.6 | Helped |
| add audio to vision-only (no pretraining) (ablation) | VAST baseline · YouCook2 captioning | CIDEr: 77.0 -> 74.1 | Hurt |
| pretrain with V+S only but finetune on audio-important benchmark (ablation) | VAST · VALOR-32K retrieval | Recall@1: 56.5 -> 48.2 | Hurt |
| add modality grouping during pretraining (ablation) | VAST · VALOR-32K retrieval | Recall@1: 58.8 -> 62.2 | Helped |
| add modality grouping during pretraining (ablation) | VAST · MSRVTT retrieval | Recall@1: 53.0 -> 55.3 | Helped |
| use VAST-27M vision captions instead of WebVid2.5M for V-T pretraining | VAST V-T model · MSVD retrieval | Recall@1: 43.1 -> 47.7 | Helped |
| use VAST-27M subtitles as captions instead of vision captions (ablation) | V-T pretraining · MSVD retrieval | Recall@1: 47.7 -> 37.3 | Hurt |
| use VAST-27M audio captions instead of WavCaps | A-T pretraining · Clothov2 retrieval | Recall@1: 22.0 -> 23.1 | Helped |
| use VAST-27M instead of VALOR-1M for pretraining | VAST OMV-OMC · VALOR-32K retrieval | Recall@1: 70.8 -> 58.8 | Hurt |
Closest papers by meaning
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.90)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.90)
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.89)
- OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning (NeurIPS 2026 · similarity 0.89)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.88)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.