Research map / Vision-language and multimodal / Cross-modal learning
EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
EquiAV adds equivariant transformation prediction to audio-visual contrastive learning and uses the centroid of attention-predicted equivariant embeddings as the inter-modal representation, reaching SOTA on AudioSet, VGGSound, retrieval and single-modal benchmarks.
Problem
Data augmentation is hard to use in audio-visual contrastive learning because augmentations can break the correspondence between paired audio and visual inputs.
Approach
Use a shared attention-based transformation predictor to map original embeddings plus augmentation vectors to equivariant embeddings; average many such embeddings (a centroid approximating the mean over augmented representations) for inter-modal contrastive learning, which avoids encoding augmented inputs and reduces the harm of augmentations on correspondence.
Key results
- EquiAV reaches 54.6 mAP on AudioSet-2M audio-visual fine-tuning, versus 53.3 for MAViL, and 67.1% on VGGSound audio-visual, tied with MAViL.
- On zero-shot retrieval over AudioSet, EquiAV reaches 30.1 V2A R@1 versus 18.8 for CAV-MAE.
- Using centroids of equivariant embeddings gives 46.6 mAP audio-visual fine-tuning on AudioSet-20K versus 43.9 when using augmented-pair embeddings; ablation shows the attention predictor's similarity 0.90/0.92 beats linear and hypernetwork predictors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace invariant intra-modal learning with equivariant learning (ablation) | EquiAV using original inter-modal embeddings · AudioSet zero-shot retrieval | V2A R@1: 25.2 -> 28.7 | Helped |
| use augmented embeddings rather than original embeddings for inter-modal learning (ablation) | EquiAV · AudioSet zero-shot retrieval | V2A R@1: 28.7 -> 28.1 | Hurt |
| use the centroid of equivariant embeddings for inter-modal learning (ablation) | EquiAV · AudioSet zero-shot retrieval | V2A R@1: 28.7 -> 30.1 | Helped |
| increase centroid representations from 1 to 16 (ablation) | EquiAV · AudioSet zero-shot retrieval | V2A R@1: 27.8 -> 30.1 | Helped |
| replace a linear transformation predictor with an attention-based predictor (ablation) | EquiAV · AudioSet zero-shot retrieval | V2A R@1: 28.2 -> 30.1 | Helped |
| replace a hypernetwork transformation predictor with an attention-based predictor (ablation) | EquiAV · AudioSet zero-shot retrieval | V2A R@1: 29.6 -> 30.1 | Helped |
| use joint training instead of two-stage training (ablation) | EquiAV · AudioSet zero-shot retrieval | V2A R@1: 27.0 -> 30.1 | Helped |
| use joint training instead of alternating iterative training (ablation) | EquiAV · AudioSet zero-shot retrieval | V2A R@1: 28.5 -> 30.1 | Helped |
| include the positive pair in the intra-modal loss denominator (ablation) | EquiAV · AudioSet zero-shot retrieval | V2A R@1: 21.9 -> 30.1 | Helped |
| set audio and visual intra-modal loss weights equally to the inter-modal weight (ablation) | EquiAV · AudioSet-20K | A-V mAP: 45.5 -> 46.6 | Helped |
| train with audio-only intra-modal self-supervision by setting the visual intra-modal weight to 0 (ablation) | EquiAV · AudioSet-20K | A mAP: 42.4 -> 42.9 | Helped |
| train with audio-only intra-modal self-supervision by setting the visual intra-modal weight to 0 (ablation) | EquiAV · AudioSet-20K | A-V mAP: 46.6 -> 43.4 | Hurt |
Surprising findings
- Using augmented-input embeddings directly in inter-modal contrastive learning slightly hurts performance versus original embeddings, indicating augmentations disrupt audio-visual correspondence.
Limitations
- Augmentation options for audio are limited compared with images, so audio augmentations are mostly visual-style transforms applied to spectrograms.
- Centroid still has variance and is an approximation of the true mean over augmentations.
- Evaluated mainly on AudioSet and VGGSound; generalization to other multimodal domains such as vision-language is left as future work.
Experimental setup
Datasets: AudioSet-2M AudioSet-20K VGGSound ESC-50 SPC-v1 FSD-50K UCF101 HMDB51 Kinetics400 MSR-VTT
Benchmarks: AudioSet-20K AudioSet-2M VGGSound ESC-50 SPC-v1 FSD-50K UCF101 HMDB51 Kinetics400 MSR-VTT
Metrics: mAP accuracy R@1 R@5 R@10
Baselines: CAV-MAE MAViL MBT AudiovisualMAE Perceiver Attn AV GBlend XKD CAV-MAE-Scale++ AVLnet XDC MMV
Builds on: EquiMod MAE CAV-MAE NT-Xent
Closest papers by meaning
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.88)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.87)
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.86)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.86)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.85)
- $\texttt{AVROBUSTBENCH}$: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.