Research map / Vision-language and multimodal / Cross-modal learning
Contrastive Audio-Visual Masked Autoencoder
Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, James R. Glass
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add masked data modeling reconstruction loss to contrastive audio-visual learning (CAV-MAE vs CAV) (ablation) | CAV-MAE · AudioSet-20K | mAP (A-V): +2.0 mAP (38.5 to 40.5) | Improved |
| add contrastive loss to masked autoencoder (CAV-MAE vs AV-MAE) (ablation) | CAV-MAE · AudioSet-20K | mAP (A-V): +3.1 mAP (37.4 to 40.5) | Improved |
| add contrastive loss to masked autoencoder (CAV-MAE vs AV-MAE) (ablation) | CAV-MAE · Visual→Audio retrieval on VGGSound | R@1: 0.1 to 14.7 R@1 | Improved |
| add modality-specific encoders before joint encoder (AV-MAE vs Vanilla AV-MAE) (ablation) | AV-MAE · AudioSet-20K | mAP (A-V): 37.4 vs 36.5 (A-V); 33.4 vs 32.7 (A) | Improved |
| multi-modal CAV-MAE pretraining used for single-modal fine-tuning (vs Audio-MAE) | CAV-MAE · AudioSet-20K audio-only | mAP: 34.2 to 37.7 | Improved |
| scale up batch size and pretraining epochs (CAV-MAE Scale+) (ablation) | CAV-MAE · AudioSet-20K | mAP (A-V): 40.5 to 42.0 | Improved |
| increase contrastive loss weight lambda_c from 0.01 to 0.1 or decrease to 0.001 (ablation) | CAV-MAE · AudioSet-20K | mAP (A-V): 39.3 (0.1) and 38.6 (0.001) vs 40.5 | Hurt |
| pretrain on AudioSet-2M then VGGSound vs VGGSound-only or AS-2M only (ablation) | CAV-MAE · VGGSound | Accuracy: 65.9 vs 65.5 (AS-2M) and 64.2 (VS) | Improved |
| normalize prediction target (ablation) | CAV-MAE · AudioSet-20K | mAP (A-V): 40.5 vs 40.5 | No clear effect |
| use multi-modal stream output for fine-tuning vs single-modal streams (ablation) | CAV-MAE · AudioSet-20K | mAP (A-V): 42.0 MM vs 41.3 SM | Improved |
| use frame aggregation (averaging predictions over 10 frames) vs middle frame at inference (ablation) | CAV-MAE · AudioSet-20K | mAP (A-V): 40.9 to 42.0 | Improved |
| use time-frequency masking at 50% ratio instead of uniform 75% (ablation) | CAV-MAE · VGGSound | Accuracy: 65.6 vs 65.4 | Improved |
Closest papers by meaning
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.92)
- EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning (ICML 2024 · similarity 0.87)
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.87)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
- Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners (ICLR 2024 · similarity 0.86)
- CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
