Research map / Vision-language and multimodal / Cross-modal learning
Audio-visual source localization
Aligns sound and visual scenes to localize and segment sounding objects.
67 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2019–2026) · an approach in Cross-modal learning, keyword cluster “audio · sound · speech”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
18 papers in 2023–24 and 37 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 6.2% to 4.7%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2019 · A Simple Baseline for Audio-Visual Scene-Aware Dialog (CVPR 2019 · 69 citations)
- 2020 · Music Gesture for Visual Sound Separation (CVPR 2020 · 202 citations)
- 2021 · Semantic Audio-Visual Navigation (CVPR 2021 · 90 citations)
- 2022 · Cross-Modal Background Suppression for Audio-Visual Event Localization (CVPR 2022 · 65 citations)
- 2023 · Vision Transformers Are Parameter-Efficient Audio-Visual Learners (CVPR 2023 · 96 citations)
- 2024 · AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection (CVPR 2024 · 76 citations)
- 2025 · Towards Open-Vocabulary Audio-Visual Event Localization (CVPR 2025 · 9 citations)
- 2026 · Do Audio-Visual Segmentation Models Truly Segment Sounding Objects? (AAAI 2026 · 1 citation)
Earliest papers here
- A Simple Baseline for Audio-Visual Scene-Aware Dialog (CVPR 2019 · 69 citations)
- Music Gesture for Visual Sound Separation (CVPR 2020 · 202 citations)
- Cross-Modal Attention Network for Temporal Inconsistent Audio-Visual Event Localization (AAAI 2020 · 106 citations)
What has been tried
49 papers here report 559 measured changes (a component added, removed or swapped, compared with a baseline): helped 80%, no clear effect 3%, hurt 15%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add the autoencoding objective, complementary masking, cross-modal fusion, and decoding | FakeAVCeleb · AP | 84.2 -> 96.7 | Helped | AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection (CVPR 2024 · 76 citations) |
| Use 300 queries instead of 1 | S4 · mIoU | 79.6 -> 83.1 | Helped | AVSegFormer: Audio-Visual Segmentation with Transformer (AAAI 2024 · 67 citations) |
| Use the encoder-prompt-decoder method instead of AV fusion | AVS-V2 · mIoU (%) | 59.69 -> 63.35 | Helped | Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · 31 citations) |
| Add audio-visual inputs to audio-only AV-RIR | SoundSpaces · WER (%) | 5.24 (+36%) -> 4.17 (+49%) | Helped | AV-RIR: Audio-Visual Room Impulse Response Estimation (CVPR 2024 · 17 citations) |
| Use 3D rather than 2D bounce-point sampling | Furnished room · C50 error (dB) | 0.57 -> 0.53 | Helped | Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark (CVPR 2024 · 14 citations) |
| Replace TPAVI cross-attention with fCA | AVSBench-Semantics and VPO (MSMI) · mIoU | 42.74 -> 47.18; 52.83 -> 58.50 | Helped | Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · 14 citations) |
| Add the AVC block to mixture patch tokens | VGGSound-Single · AP | 42.8 -> 43.9 | Helped | T-VSL: Text-Guided Visual Sound Source Localization in Mixtures (CVPR 2024 · 10 citations) |
| add the special text other | OV-AVEBench total test · Avg. | 47.0 -> 57.8 | Helped | Towards Open-Vocabulary Audio-Visual Event Localization (CVPR 2025 · 9 citations) |
| Add audio information | AV-TAU test (Overall) · GPT-Eval | 0.7190 -> 0.8103 | Helped | EchoTraffic: Enhancing Traffic Anomaly Understanding with Audio-Visual Insights (CVPR 2025 · 7 citations) |
| Add OCA loss | VGGSound-Duet test · CAP (%) | 28.8 -> 43.8 | Helped | Object-aware Sound Source Localization via Audio-Visual Scene Understanding (CVPR 2025 · 6 citations) |
| use multilingual AV-HuBERT rather than English-trained AV-HuBERT | MuAViC Es-En · ASR-BLEU | 17.63 -> 26.57 | Helped | AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation (CVPR 2024 · 6 citations) |
| replace conventional attention with proxy aggregation | AVSBench-Semantic · mIoU | 37.0 -> 47.1 | Helped | Few-Shot Incremental Learning via Foreground Aggregation and Knowledge Transfer for Audio-Visual Semantic Segmentation (AAAI 2025 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 22 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- VGGSound · 5 of 22 papers
- AudioSet · 4 of 22 papers
- AVE · 2 of 22 papers
- AVS-Bench · 2 of 22 papers
- Ref-AVS · 2 of 22 papers
- AudioCaps · 2 of 22 papers
- AVS-Objects · 2 of 22 papers
- AVS-Semantic · 2 of 22 papers
Baselines
- AVSegFormer · 6 of 22 papers
- CATR · 4 of 22 papers
- CAVP · 4 of 22 papers
- AVSBench · 3 of 22 papers
- BAVS · 3 of 22 papers
- TPAVI · 3 of 22 papers
- AVSC · 3 of 22 papers
- COMBO · 3 of 22 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 55 papers; from OpenAlex and the papers' affiliations).
- Korea Advanced Institute of Science and Technology · 5 papers
- Zhejiang University · 3 papers
- Nanyang Technological University · 2 papers
- Nanjing University · 2 papers
- The Chinese University of Hong Kong · 2 papers
- University of Science and Technology of China · 2 papers
- A*STAR · 2 papers
- Chinese Academy of Sciences · 2 papers
- University of Texas at Dallas · 2 papers
- University of Michigan · 2 papers
Use it
Download BibTeX of 36 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Audio-visual speech separation (76 papers)
- Self-supervised audio-visual representation learning (44 papers)
Elsewhere in the topic
- Multimodal fusion with missing modalities (208 papers)
- Multi-modality image fusion (194 papers)
- Medical multimodal fusion (166 papers)
- Unimodal-to-multimodal representation transfer (145 papers)
- Probabilistic embeddings for cross-modal retrieval (108 papers)
- Multimodal sentiment analysis (106 papers)
- Contrastive vision-language alignment (104 papers)
- Cross-modal hashing (45 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).