Research map / Vision-language and multimodal / Cross-modal learning
Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning
Apoorv Vyas, Heng-Jui Chang, Cheng-Fu Yang, Po-Yao Huang, Luya Gao, Julius Richter, Sanyuan Chen, Matthew Le, Piotr Dollár, Christoph Feichtenhofer, Ann Lee, Wei-Ning Hsu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add stage-2 improved captions over stage-1 captions (ablation) | PEAV-B · pre-training ablation average | avg score: 38.9 -> 41.5 | Improved |
| use data-engine captions instead of raw EnCLAP captions (ablation) | PEAV-B · pre-training ablation average | avg score: 33.1 -> 41.5 | Improved |
| use synthetic captions only instead of real only (ablation) | PEAV-B · data mixing ablation | avg score: 14.9 -> 43.0 | Improved |
| mix real and synthetic captions at 1:10 instead of synthetic only (ablation) | PEAV-B · data mixing ablation | avg score: 43.0 -> 45.4 | Improved |
| raise synthetic ratio from 1:10 to 1:30 (ablation) | PEAV-B · data mixing ablation | avg score: 45.4 -> 44.2 | No clear effect |
| scale training data from 2M to 64M (ablation) | PEAV-B · data scaling ablation | avg score: 38.4 -> 43.4 | Improved |
| scale audio encoder depth beyond 20 layers (ablation) | PEAV-B · audio encoder scaling | avg score: 44.5 -> 42.0 | Hurt |
| scale audio encoder from 8 to 20 layers (ablation) | PEAV-B · audio encoder scaling | avg score: 41.1 -> 44.5 | Improved |
| expand contrastive pairs from one audio-text pair to all eight (ablation) | PEAV-B · contrastive coverage ablation | avg score: 19.0 -> 43.2 | Improved |
| use native joint embedding instead of max over uni-modal queries (ablation) | PEAV-L · AudioCaps T+V->A | R@1: 69.8 -> 80.2 | Improved |
| add stage-2 fine-tuning with speech data (ablation) | PEAV-L · VCTK speech-to-transcript | R@1: 16.7 -> 85.6 | Improved |
| use 30 fps instead of 16 frames (ablation) | PEAV-L · video retrieval average | avg R@1: 66.9 -> 67.9 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.
- LanguageBind — LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (ICLR 2024) · better on 3 of 3 shared results
- ImageBind — ImageBind: One Embedding Space To Bind Them All (CVPR 2023) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.90)
- An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching (NeurIPS 2024 · similarity 0.89)
- Looking Similar Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning (CVPR 2024 · similarity 0.89)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.88)
- Aligned Better, Listen Better for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.88)
- Asymmetric Hierarchical Anchoring for Robust Audio–Visual Cross-Modal Generalization (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
