Research map / Video / Video-language understanding
Learning Audio-Video Modalities from Image Captions
Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, Cordelia Schmid
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain on VideoCC3M instead of HowTo100M | RGB retrieval model · MSR-VTT finetuned | R@1: 33.1 -> 35.0 | Improved |
| pretrain on VideoCC3M instead of HowTo100M | RGB retrieval model · MSR-VTT zero-shot | R@1: 8.6 -> 18.9 | Improved |
| add audio to VideoCC3M-pretrained model (ablation) | retrieval model · MSR-VTT finetuned | R@1: 35.0 -> 35.8 | Improved |
| add audio to VideoCC3M-pretrained model (ablation) | retrieval model · MSR-VTT zero-shot | R@1: 18.9 -> 19.4 | Improved |
| add VideoCC3M pretraining to RGB model | retrieval model · MSR-VTT finetuned | R@1: 30.2 -> 35.0 | Improved |
| initialise video encoder from K400 instead of ImageNet21K (ablation) | RGB video encoder · MSR-VTT | R@1: 30.2 -> 30.2 | No clear effect |
| initialise video encoder from ImageNet21K instead of scratch (ablation) | RGB video encoder · MSR-VTT | R@1: 9.4 -> 30.2 | Improved |
| add audio stream to K400-initialised video model (ablation) | retrieval model · MSR-VTT | R@1: 30.2 -> 32.3 | Improved |
| increase mined clip length t from 3s to 10s (ablation) | VideoCC3M mining · MSR-VTT zero-shot | R@1: 16.4 -> 18.9 | Improved |
| increase frame stride from 2 to 14 (longer clip coverage) (ablation) | RGB retrieval model · MSR-VTT | R@1: 24.1 -> 27.3 | Improved |
| initialise audio encoder from VGGSound instead of ImageNet21K (ablation) | audio retrieval model · AudioCaps | R@1: 30.2 -> 32.0 | Improved |
| pretrain audio model on VideoCC3M | audio retrieval model · AudioCaps (audio only) | R@1: 32.0 -> 35.5 | Improved |
Closest papers by meaning
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (CVPR 2024 · similarity 0.89)
- Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions (CVPR 2025 · similarity 0.87)
- An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching (NeurIPS 2024 · similarity 0.86)
- Vript: A Video Is Worth Thousands of Words (NeurIPS 2024 · similarity 0.86)
- Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning (AAAI 2025 · similarity 0.86)
- AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
