Research map / Vision-language and multimodal / Cross-modal learning
The Right To Talk: An Audio-Visual Transformer Approach
Thanh-Dat Truong, Chi Nhan Duong, The De Vu, Hoang Anh Pham, Bhiksha Raj, Ngan Le, Khoa Luu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-domain alignment loss L_align (ablation) | Audio-Visual Transformer (block attention) · LRS2 separation 1S+N | SDR (dB): 14.4 -> 15.8 | Improved |
| add cross-domain alignment loss L_align (ablation) | Audio-Visual Transformer (block attention) · LRS2 separation 2S | SDR (dB): 9.8 -> 10.3 | Improved |
| add cross-domain alignment loss L_align (ablation) | Audio-Visual Transformer (block attention) · LRS2 separation 2S+N | SDR (dB): 7.0 -> 7.2 | Improved |
| add cross-domain alignment loss L_align (ablation) | Audio-Visual Transformer (block attention) · LRS2 separation 2S+N | PESQ: 2.5 -> 2.5 | No clear effect |
| add across-segment temporal attention (ablation) | Audio-Visual Transformer (block attention) · Columbia (Avg) cooperative localization | F1 (%): 92.7% -> 93.4% | Improved |
| add across-segment temporal attention (ablation) | Audio-Visual Transformer (speaker attention) · Columbia (Avg) cooperative localization | F1 (%): 93.8% -> 94.9% | Improved |
| add across-segment temporal attention (ablation) | Audio-Visual Transformer (block attention) · LRS2 separation 2S | SDR (dB): 10.3 -> 11.5 | Improved |
| add across-segment temporal attention (ablation) | Audio-Visual Transformer (block attention) · LRS2 separation 2S+N | SDR (dB): 7.2 -> 8.1 | Improved |
| add across-segment temporal attention (ablation) | Audio-only variant · LRS2 separation 2S | SDR (dB): 9.1 -> 9.4 | Improved |
| add visual stream to the audio-only model (ablation) | Audio-Visual Transformer · LRS2 separation 2S | SDR (dB): 9.1 -> 10.3 | Improved |
| use speaker (face-detection) regions instead of block decomposition (ablation) | Audio-Visual Transformer · LRS2 separation 1S+N | SDR (dB): 15.8 -> 16.5 | Improved |
| use speaker regions instead of block decomposition (ablation) | Audio-Visual Transformer · Columbia (Avg) cooperative localization | F1 (%): 92.7% -> 93.8% | Improved |
Closest papers by meaning
- The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective (CVPR 2024 · similarity 0.86)
- Multi-speaker Attention Alignment for Multimodal Social Interaction (CVPR 2026 · similarity 0.85)
- Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs (NeurIPS 2026 · similarity 0.85)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.84)
- Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics (ICLR 2025 · similarity 0.84)
- LoCoNet: Long-Short Context Network for Active Speaker Detection (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
