Research map / Vision-language and multimodal / Cross-modal learning
Audio-visual speech separation
Uses visual cues such as lips and cross-modal prompts to separate speech and adapt pretrained models.
76 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Cross-modal learning, keyword cluster “audio · sound · speech”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
12 papers in 2023–24 and 40 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 4.1% to 5.1%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Temporal Multimodal Learning in Audiovisual Speech Recognition (CVPR 2016 · 113 citations)
- 2018 · Multi-attention Recurrent Network for Human Communication Comprehension (AAAI 2018 · 516 citations)
- 2020 · M3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues (AAAI 2020 · 258 citations)
- 2021 · VisualVoice: Audio-Visual Speech Separation With Cross-Modal Consistency (CVPR 2021 · 165 citations)
- 2022 · Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation (CVPR 2022 · 128 citations)
- 2023 · VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset (NeurIPS 2023 · 28 citations)
- 2024 · V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · 36 citations)
- 2025 · Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content (AAAI 2025 · 49 citations)
- 2026 · WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM (ICLR 2026 · 2 citations)
Earliest papers here
- Temporal Multimodal Learning in Audiovisual Speech Recognition (CVPR 2016 · 113 citations)
- Look, Listen and Learn — A Multimodal LSTM for Speaker Identification (AAAI 2016 · 73 citations)
- Multi-attention Recurrent Network for Human Communication Comprehension (AAAI 2018 · 516 citations)
What has been tried
44 papers here report 510 measured changes (a component added, removed or swapped, compared with a baseline): helped 79%, no clear effect 4%, hurt 15%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add emotional descriptions (AED and VED) | MOSI · Acc-7 | 43.00 -> 46.21 | Helped | Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content (AAAI 2025 · 49 citations) |
| add the V2A-Mapper rather than pass visual features directly to AudioLDM | VGGSound · FD | 72.527 -> 24.168 | Helped | V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · 36 citations) |
| Remove canonical volume loss L_n^CV | MEAD-driven cross-reenactment · UMTN | 45.4 -> 9.2 | Hurt | EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars (CVPR 2024 · 28 citations) |
| combine audio and visual control instead of audio-only control | HDTF · FID | 34.233 -> 31.497 | Helped | MegActor-Sigma: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformer (AAAI 2025 · 9 citations) |
| add the audio classification loss La | KoDF · Acc. | 73.5 -> 92.0 | Helped | Multi-modal Deepfake Detection via Multi-task Audio-Visual Prompt Learning (AAAI 2025 · 6 citations) |
| replace BERT text encoding with Trimodal-style text feature extraction | TED Gesture · FGD | 1.406 -> 1.955 | Hurt | HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generation (CVPR 2025 · 5 citations) |
| Add audio to text input | MMPsy-Anxiety · UAR | 0.74 -> 0.76 | Helped | Mental-Perceiver: Audio-Textual Multi-Modal Learning for Estimating Mental Disorders (AAAI 2025 · 4 citations) |
| Combine the contrastive and generative pretraining objectives | Respiratory benchmark tasks · performance | No clear effect | Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarking (NeurIPS 2024 · 4 citations) | |
| Add dominance-aware masking | AVHBench validation · accuracy (%) | 74.15 -> 79.02 | Helped | AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding (NeurIPS 2025 · 1 citation) |
| Increase the decoder size from Qwen2 0.5B to Qwen2 72B | Language Dominance · HR | 55.0 -> 75.5 | Helped | The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio (NeurIPS 2025 · 1 citation) |
| Add IAL losses LG/LD | MUSIC · SDR | 8.08 -> 10.22 | Helped | Independency Adversarial Learning for Cross-Modal Sound Separation (AAAI 2024 · 1 citation) |
| Use CAF instead of adapted CTCNet fusion | LRS2-2Mix · SI-SNRi | 11.3 -> 11.7 | Helped | RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation (ICLR 2024 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 18 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- AudioCaps · 3 of 18 papers
- VGGSound · 3 of 18 papers
- Clotho · 2 of 18 papers
- OmniBench · 2 of 18 papers
- WorldSense · 2 of 18 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 52 papers; from OpenAlex and the papers' affiliations).
- Chinese Academy of Sciences · 2 papers
- Monash University · 2 papers
- Tsinghua University · 2 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Audio-visual source localization (67 papers)
- Self-supervised audio-visual representation learning (44 papers)
Elsewhere in the topic
- Multimodal fusion with missing modalities (208 papers)
- Multi-modality image fusion (194 papers)
- Medical multimodal fusion (166 papers)
- Unimodal-to-multimodal representation transfer (145 papers)
- Probabilistic embeddings for cross-modal retrieval (108 papers)
- Multimodal sentiment analysis (106 papers)
- Contrastive vision-language alignment (104 papers)
- Cross-modal hashing (45 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).