Research map / Video / Video segmentation and tracking
MAViL: Masked Audio-Video Learners
Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Haoqi Fan, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add masked contrastive learning (inter+intra) to joint AV-MAE (ablation) | MAViL stage-1 · AS-20K audio | mAP: 36.8 -> 39.0 | Improved |
| add masked contrastive learning (inter+intra) (ablation) | MAViL stage-1 · AS-20K video | mAP: 17.7 -> 22.2 | Improved |
| add intra-modal contrast on top of inter-modal only (ablation) | MAViL stage-1 · AS-20K audio | mAP: 38.4 -> 39.0 | Improved |
| add audio-video fusion encoder (ablation) | uni-modal MAE · AS-20K audio | mAP: 36.4 -> 36.8 | Improved |
| MBT instead of vanilla Transformer fusion encoder (ablation) | fusion encoder · AS-20K | mAP: 36.8 -> 36.8 (audio) | No clear effect |
| stage-2 self-training to predict teacher contextualized representations instead of raw targets (ablation) | MAViL · AS-20K audio | mAP: 39.0 -> 41.8 | Improved |
| use multimodal fusion output of teacher as target vs uni-modal output (ablation) | MAViL stage-2 · AS-20K audio | mAP: 40.2 -> 40.7 | Improved |
| increase self-training iterations from 1 to 3 (ablation) | MAViL stage-2 · AS-20K audio | mAP: 40.7 -> 41.8 | Improved |
| raise masking ratio from 40% to 80% (ablation) | MAViL stage-1 · AS-20K audio | mAP: 35.6 -> 36.8 | Improved |
| raise masking ratio from 80% to 90% (ablation) | MAViL stage-1 · AS-20K video | mAP: 17.7 -> 17.5 | Hurt |
| initialize visual backbone with ImageNet SSL MAE (ablation) | MAViL · AS-20K video | mAP: 23.7 -> 24.8 | Improved |
| increase encoder size ViT-B to ViT-L (ablation) | MAViL · AS-20K audio | mAP: 41.8 -> 42.2 | Improved |
Closest papers by meaning
- Audiovisual Masked Autoencoders (ICCV 2023 · similarity 0.88)
- TriBERT: Human-centric Audio-visual Representation Learning (NeurIPS 2021 · similarity 0.87)
- Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.86)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.86)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.86)
- AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
