Research map / Vision-language and multimodal / Cross-modal learning
Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeepAVFusion, an audio-visual early-fusion transformer with factorized local audio-visual interactions, pre-trained via multi-modal masked autoencoding, and evaluated on classification, separation, and segmentation.
Problem
Early fusion audio-visual architectures are under-explored and hard to train, since prior masked-modeling and contrastive methods mostly use late fusion, and dense local audio-visual interactions are computationally expensive.
Approach
Fusion tokens cross-attend to pairwise linear audio-visual interactions over local tokens; to reduce cost, audio and visual tokens are first summarized into small sets of aggregation tokens before pairwise interactions, and the whole early-fusion encoder is trained with an audio-visual masked autoencoding objective initialized from unimodal MAEs.
Key results
- On VGGSound-Music source separation, DeepAVFusion reaches 5.79 SDR, outperforming OneAVM (3.67 SDR) and other baselines.
- On AVSBench S4, DeepAVFusion achieves 89.94 mIoU versus 81.74 for MMVAE.
- On VGGSound-Music linear probing, DeepAVFusion reaches 65.32% accuracy versus 59.18% for CAV-MAE; factorized interactions give a 700x reduction in interactions while keeping the benefit of dense interactions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add unimodal pre-training (ablation) | DeepAVFusion · VGGSound | Linear accuracy: 44.36 -> 53.08 | Helped |
| add local audio-visual interactions instead of token fusion (ablation) | DeepAVFusion · VGGSound | Linear accuracy: 51.19 -> 53.08 | Helped |
| use early rather than mid fusion (ablation) | DeepAVFusion · VGGSound | Linear accuracy: 51.21 -> 53.08 | Helped |
| use early rather than late fusion (ablation) | DeepAVFusion · VGGSound | Linear accuracy: 46.52 -> 53.08 | Helped |
| use early fusion rather than no fusion (ablation) | DeepAVFusion · VGGSound | Linear accuracy: 39.67 -> 53.08 | Helped |
| factorize dense audio-visual interactions (ablation) | DeepAVFusion early-fusion encoder · inference efficiency | Speed (video/sec): 4.3 -> 12.8 | Helped |
| factorize dense audio-visual interactions (ablation) | DeepAVFusion early-fusion encoder · inference efficiency | Memory (GB): 31.6 -> 13.1 | Helped |
| increase fusion layers from 1 to 12 (ablation) | DeepAVFusion · VGGSound | Linear accuracy: 46.52 -> 53.08 | Helped |
| increase fusion tokens from 1 to 16 (ablation) | DeepAVFusion · VGGSound | Linear accuracy: 51.86 -> 53.08 | Helped |
| increase aggregation tokens from 4 to 8 (ablation) | DeepAVFusion · VGGSound | Linear accuracy: 51.78 -> 53.08 | Helped |
Surprising findings
- Fusion tokens yield higher-level semantic representations than unimodal tokens despite the MAE objective favoring low-level detail.
- Dense interactions gave a 3.3x throughput slowdown and 3.1x memory increase, which factorized interactions largely recover.
Limitations
- Visual branch uses single frames from image models rather than video, so video modeling is not addressed.
- Comparisons to prior work use differing audio-visual encoder architectures, making direct comparison hard.
Experimental setup
Datasets: VGGSound AudioSet ImageNet MUSIC VGGSound-Instruments VGGSound-Music AVSBench Flickr-SoundNet
Benchmarks: VGGSound-Music VGGSound-All AudioSet MUSIC VGGSound-Instruments AVSBench Flickr-SoundNet
Metrics: SDR SIR SAR mIoU F1 classification accuracy average precision AUC nearest neighbor accuracy
Baselines: NMF RPCA Sound-of-Pixels MP-Net CCoL OneAVM AVS LAVISH MMVAE MAE AudioMAE CAV-MAE
Builds on: Masked Autoencoders (MAE) AudioMAE CAV-MAE MBT
Closest papers by meaning
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.89)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.89)
- Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.88)
- Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization (NeurIPS 2025 · similarity 0.88)
- Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation (CVPR 2025 · similarity 0.87)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.