Research map / Vision-language and multimodal / Cross-modal learning
CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CrossMAE, an audio-visual masked autoencoder pre-training method with cross-conditioned and cross-embedding reconstruction plus contrastive loss, aimed at region-aware alignment for dense prediction tasks.
Problem
Existing audio-visual pre-training (e.g., CAV-MAE, AudioCLIP) focuses on global alignment, neglecting fine-grained region and positional features, so it performs poorly on dense prediction tasks such as audio-visual source localization.
Approach
Dual-level masked autoencoding across modalities: pixel-level reconstruction of each modality conditioned on the other's top-attended (Attentive) tokens, and embedding-level reconstruction of the other modality's full features via DETR-style Learnable Queries, combined with audio-visual contrastive loss.
Key results
- CrossMAE reaches 48.2 A-V mAP on AudioSet-20K (vs 42.0 for CAV-MAE) and 67.0 A-V accuracy on VGGSound-200K (vs 65.5 for CAV-MAE).
- On Flickr-SoundNet source localization, CrossMAE achieves 73.20 CIoU and 42.61 AUC, improving AUC by 11.78% over the CL-AV-style baselines as stated in the paper.
- Audio-to-visual R@1 retrieval on Flickr-SoundNet reaches 16.8 vs 12.2 for CAV-MAE; performance improves with larger batch size and ViT-L/16 backbone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Cross-Conditioned Reconstruction to contrastive learning (ablation) | CrossMAE pre-training · AudioSet-20K | A-V mAP: 38.2 -> 44.9 | Helped |
| Add Cross-Embedding Reconstruction to contrastive learning (ablation) | CrossMAE pre-training · AudioSet-20K | A-V mAP: 38.2 -> 47.7 | Helped |
| Add both Cross-Conditioned and Cross-Embedding Reconstruction (ablation) | CrossMAE pre-training · AudioSet-20K | A-V mAP: 38.2 -> 48.2 | Helped |
| Add both region-reconstruction objectives rather than contrastive learning alone (ablation) | CrossMAE · AudioSet-2M | A-V mAP: 48.2 -> 55.3 | Helped |
| Add both region-reconstruction objectives rather than contrastive learning alone (ablation) | CrossMAE · VGGSound-200K | A-V accuracy: 57.1 -> 67.0 | Helped |
| Add Cross-Conditioned Reconstruction to contrastive learning (ablation) | CrossMAE pre-training · Flickr-SoundNet | CIoU: 65.60 -> 68.53 | Helped |
| Add Cross-Embedding Reconstruction to contrastive learning (ablation) | CrossMAE pre-training · Flickr-SoundNet | CIoU: 65.60 -> 69.11 | Helped |
| Add self-modality MAE reconstruction to contrastive learning (ablation) | CrossMAE pre-training · Flickr-SoundNet | CIoU: 65.60 -> 67.20 | Helped |
| Replace Attentive Tokens with zero vectors (ablation) | CL-CrossEmbMAE · Flickr-SoundNet audio-to-visual retrieval | R@1: 22.8 -> 19.2 | Hurt |
| Replace Attentive Tokens with random tokens (ablation) | CL-CrossEmbMAE · Flickr-SoundNet audio-to-visual retrieval | R@1: 22.8 -> 19.6 | Hurt |
| Replace Learnable Queries with empty queries (ablation) | CL-CrossCdtMAE · Flickr-SoundNet audio-to-visual retrieval | R@1: 22.8 -> 17.6 | Hurt |
| Use Learnable Queries instead of Attentive Tokens as the Cross-Embedding Reconstruction input (ablation) | CL-CrossCdtMAE · Flickr-SoundNet audio-to-visual retrieval | R@1: 15.8 -> 22.8 | Helped |
Surprising findings
- Using tokens from the other modality as queries hurts performance because they leak too much reconstruction-target information, while empty or learnable queries help.
- Random tokens for conditioning improve retrieval only modestly and empty vectors give no gain, showing attentive token selection matters.
Limitations
- Relies on pretraining with AudioSet-2M and ImageNet initialization, with large-batch gains that are not fully characterized.
- Dense prediction evaluation limited to source localization on two datasets in the main text.
Experimental setup
Datasets: AudioSet-2M AudioSet-20K VGGSound-200K Flickr-SoundNet VGG-SoundSource
Benchmarks: AudioSet-20K AudioSet-2M VGGSound-200K Flickr-SoundNet VGG-SoundSource
Metrics: mAP accuracy R@1 R@5 R@10 CIoU AUC
Baselines: Audio-CLIP Perceiver Attn AV MBT CAV-MAE AV-MAE MAViL CL-AV SelfMAE CL-SelfMAE
Builds on: masked autoencoders CAV-MAE AudioMAE DETR AudioCLIP
Closest papers by meaning
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.88)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.87)
- Asymmetric Hierarchical Anchoring for Robust Audio–Visual Cross-Modal Generalization (NeurIPS 2026 · similarity 0.87)
- X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation (NeurIPS 2026 · similarity 0.87)
- Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (NeurIPS 2026 · similarity 0.87)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.