Research map / Vision-language and multimodal / Cross-modal learning
CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends CAV-MAE with frame-aligned audio segment tokens, separate global and register tokens for contrastive vs. reconstruction objectives, achieving SOTA zero-shot audio-visual retrieval, linear-probe classification, and sound-prompted segmentation.
Problem
Audio-visual self-supervised models align video and audio using a single global audio representation, missing fine-grained temporal correspondence, and jointly optimizing contrastive and reconstruction objectives on the same tokens creates conflicts.
Approach
Treat audio as a sequence of spectrogram segments temporally aligned with individual video frames, add dedicated global tokens for the contrastive loss so patch tokens focus on reconstruction, and add register tokens to reduce semantic load on patch tokens, all within the CAV-MAE framework.
Key results
- Zero-shot retrieval on AudioSet Visual-to-Audio R@1 of 35.2 vs 22.1 for ImageBind and 16.1 for CAV-MAE.
- Linear probing reaches 30.5 mAP on AudioSet-20K and 52.7% accuracy on VGGSound using AS-2M pretraining, beating CAV-MAE variants and MaViL.
- Sound-prompted segmentation on ADE20K Sound reaches 22.7 mIoU, on par with CAV-MAE and ImageBind but below DenseAV (24.2 with DinoV2+LoRA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use fine-grained audio segments temporally aligned with video frames (ablation) | CAV-MAE Sync · AudioSet and VGGSound retrieval | R@1: CAV-MAEScale+++: 30.1; CAV-MAE Sync: 35.2 (AudioSet), 27.9 (VGGSound) | Helped |
| use 16 temporal tokens with diagonal similarity (ablation) | CAV-MAE · AudioSet Eval Subset | R@1: 19.7 -> 23.9 | Helped |
| increase batch size to 512 and set εc=0.1 (ablation) | CAV-MAE · AudioSet Eval Subset | R@1: 23.9 -> 30.1 | Helped |
| increase frame sampling from 10 to 16 in both training and evaluation (ablation) | CAV-MAE Sync · AudioSet Eval Subset | R@1: 31.2 -> 35.2 | Helped |
| add register tokens (ablation) | CAV-MAE Sync · ADE20K Sound | mIoU: 20.9 -> 22.9 | Helped |
| add a global token (ablation) | CAV-MAE Sync · AudioSet Eval Subset | R@1: 30.1 -> 35.2 | Helped |
| add a global token (ablation) | CAV-MAE Sync · ADE20K Sound | mIoU: 20.7 -> 22.8 | Helped |
| set masking ratio to 0.6 instead of 0.75 (ablation) | CAV-MAE Sync · AudioSet Eval Subset | R@1: 35.2 -> 39.1 | Helped |
| set masking ratio to 0.6 instead of 0.75 (ablation) | CAV-MAE Sync · ADE20K Sound | mIoU: 22.8 -> 19.1 | Hurt |
| set masking ratio to 0.9 instead of 0.75 (ablation) | CAV-MAE Sync · AudioSet Eval Subset | R@1: 35.2 -> 21.5 | Hurt |
| set masking ratio to 0.9 instead of 0.75 (ablation) | CAV-MAE Sync · ADE20K Sound | mIoU: 22.8 -> 23.4 | Helped |
| use a multi-ratio masking strategy (ablation) | CAV-MAE Sync · AudioSet Eval Subset | R@1: 35.2 -> 27.3 | Hurt |
Surprising findings
- Using 3-4 second audio segments outperforms standard 10-second segments for zero-shot retrieval while cutting audio token count.
- Applying a multi-ratio masking strategy from AVSiam led to worse performance across all metrics.
Limitations
- Segmentation performance (22.7 mIoU) trails the stronger DenseAV baseline (24.2 mIoU with DinoV2+LoRA).
- Audio segments shorter than 3 seconds degrade performance, indicating a limit to the approach.
- Ablation and hyperparameter tuning use a small set of configurations; the paper cites a trade-off between retrieval and localization across masking ratios.
Experimental setup
Datasets: AudioSet-2M AudioSet-20K VGGSound ADE20K Sound
Benchmarks: AudioSet VGGSound AudioSet-20K ADE20K Sound
Metrics: Recall@1 Recall@5 Recall@10 mAP accuracy mIoU
Baselines: CAV-MAE CAV-MAEScale+ ImageBind AVSiam VAB-Encodec LanguageBind MaViL DenseAV DAVENet
Builds on: CAV-MAE Vision Transformer registers AVMAE
Closest papers by meaning
- CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training (CVPR 2024 · similarity 0.88)
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.88)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.88)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.88)
- EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning (ICML 2024 · similarity 0.87)
- Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.