Research map / Vision-language and multimodal / Cross-modal learning
Self-supervised audio-visual representation learning
Learns audio-visual features from correspondence and synchronization via contrastive and distillation objectives.
44 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in Cross-modal learning, keyword cluster “audio · sound · speech”
State History Benchmarks Tried Preprints Who Export
Where it stands
Quiet None of the tracked benchmarks has a report from 2025–26.
18 papers in 2023–24 and 12 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 6.2% to 1.5%.
No longer reported by papers here: ESC-50 (accuracy, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · Deep Multimodal Clustering for Unsupervised Audiovisual Learning (CVPR 2019 · 210 citations)
- 2020 · Self-Supervised MultiModal Versatile Networks (NeurIPS 2020 · 193 citations)
- 2021 · Audio-Visual Instance Discrimination with Cross-Modal Agreement (CVPR 2021 · 214 citations)
- 2022 · Balanced Multimodal Learning via On-the-Fly Gradient Modulation (CVPR 2022 · 336 citations)
- 2023 · Collecting Cross-Modal Presence-Absence Evidence for Weakly-Supervised Audio-Visual Event Perception (CVPR 2023 · 42 citations)
- 2024 · Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling (CVPR 2024 · 25 citations)
- 2025 · Language-Guided Audio-Visual Learning for Long-Term Sports Assessment (CVPR 2025 · 15 citations)
- 2026 · OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attention (ICML 2026 · 1 citation)
Earliest papers here
- Deep Multimodal Clustering for Unsupervised Audiovisual Learning (CVPR 2019 · 210 citations)
- Self-Supervised MultiModal Versatile Networks (NeurIPS 2020 · 193 citations)
- Audio-Visual Instance Discrimination with Cross-Modal Agreement (CVPR 2021 · 214 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
ESC-50 · accuracy (higher is better)
4 papers, 2021–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 88.3 | Enhancing Audio-Visual Association with Self-Supervised Curriculum Learning (AAAI 2021 · 22 citations) |
| 2024 | 96 | EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning (ICML 2024 · 1 citation) |
What has been tried
35 papers here report 410 measured changes (a component added, removed or swapped, compared with a baseline): helped 71%, no clear effect 5%, hurt 20%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add unimodal pre-training | VGGSound · Linear accuracy | 44.36 -> 53.08 | Helped | Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling (CVPR 2024 · 25 citations) |
| add cross-modal distillation and domain alignment instead of video-only masked reconstruction | UCF101, HMDB51, Kinetics-Sound, and Kinetics400 · top-1 accuracy | 8.6%; 8.2%; 13.9%; 15.7% | Helped | XKD: Cross-Modal Knowledge Distillation with Domain Alignment for Video Representation Learning (AAAI 2024 · 21 citations) |
| Add S²CE | FS1000 (TES) · Spearman correlation | 0.835 -> 0.848 | Helped | Language-Guided Audio-Visual Learning for Long-Term Sports Assessment (CVPR 2025 · 15 citations) |
| Add Cross-Conditioned Reconstruction to contrastive learning | AudioSet-20K · A-V mAP | 38.2 -> 44.9 | Helped | CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training (CVPR 2024 · 12 citations) |
| Add 900 hours of unlabeled single-channel audio for joint training | MISP2021-AVSR far-field C4 · CER (%) | 71.03 -> 68.24 | Helped | Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation (AAAI 2024 · 10 citations) |
| Add multimodal consensus loss (stage two) to contrastive MC3 training | Ego4D sounding action discovery · ROC AUC (AV) | 0.598 vs 0.563 without consensus loss | Helped | SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos (CVPR 2024 · 9 citations) |
| Add a second unimodal adapter while keeping one cross-modal adapter | AVSBench MS3 · MF | 67.5 -> 68.1 | Helped | Mixtures of Experts for Audio-Visual Learning (NeurIPS 2024 · 9 citations) |
| add event-specific class feature decoupling | LLP test · Avg. F-score | 63.0 -> 64.2 | Helped | Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing (AAAI 2025 · 8 citations) |
| Progress from Stage 1 to full ES3 Stage 3, adding video-unique and joint audio-visual learning | LRS2-BBC low-resource · VSR WER | 58.3 (57.3) -> 48.1 (46.9) | Helped | ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations (CVPR 2024 · 8 citations) |
| use fine-grained audio segments temporally aligned with video frames | AudioSet and VGGSound retrieval · R@1 | CAV-MAEScale+++: 30.1; CAV-MAE Sync: 35.2 (AudioSet), 27.9 (VGGSound) | Helped | CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · 7 citations) |
| remove disentanglement loss and head max-pooling | Sound/language disentanglement · Pred. Dis. | 64.1% -> 99.9% | Hurt | Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language (CVPR 2024 · 7 citations) |
| Add dub augmentation to A.1 | HEAR sound and scene classification and non-semantic speech tasks · accuracy / mean average precision | Helped | Looking Similar Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning (CVPR 2024 · 4 citations) |
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 30 papers; from OpenAlex and the papers' affiliations).
- Massachusetts Institute of Technology · 4 papers
- Meta · 3 papers
- Vector Institute · 2 papers
- Queen’s University · 2 papers
- University of Science and Technology of China · 2 papers
- University of Chinese Academy of Sciences · 2 papers
- MIT-IBM Watson AI Lab · 2 papers
- Microsoft · 2 papers
- IBM · 2 papers
Use it
Download BibTeX of 36 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Audio-visual source localization (67 papers)
- Audio-visual speech separation (76 papers)
Elsewhere in the topic
- Multimodal fusion with missing modalities (208 papers)
- Multi-modality image fusion (194 papers)
- Medical multimodal fusion (166 papers)
- Unimodal-to-multimodal representation transfer (145 papers)
- Probabilistic embeddings for cross-modal retrieval (108 papers)
- Multimodal sentiment analysis (106 papers)
- Contrastive vision-language alignment (104 papers)
- Cross-modal hashing (45 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).