Research map / Vision-language and multimodal / Cross-modal learning
Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes
Wendi Sang, Kai Li, Yifan Li, Jianqiang Huang, Xiaolin Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Targets streaming audio-visual target speaker extraction for real-world acoustic scenes with lightweight, edge-deployable causal models.
Problem
Current causal streaming audio-visual target speaker extraction (AVTSE) systems are evaluated on benchmarks with exactly two speakers that ignore near/far-field interference, reverberation, and sudden noise, and high-performing causal architectures are too heavy for edge deployment.
Approach
Abstract is truncated; the paper pairs more realistic multi-speaker, reverberant, noisy evaluation conditions with a lightweight causal AVTSE architecture for edge deployment. Specific architectural details are not stated in the available text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase separator downsampling stages from one to three (ablation) | Falcon · RealSSA-Sim | SI-SNRi: 7.1 -> 7.7 | Improved |
| add encoder-space speaker contrastive loss (ablation) | Falcon · RealSSA-Sim | SI-SNRi: 7.3 -> 7.7 | Improved |
| add encoder-space speaker contrastive loss | SwiftNet-12 · RealSSA-Sim | SI-SNRi: 7.3 -> 7.5 | Improved |
| add encoder-space speaker contrastive loss | Dolphin · RealSSA-Sim | SI-SNRi: 6.7 -> 7.0 | Improved |
| anneal contrastive temperature tau from 0 to 1 instead of fixed tau=0 (ablation) | Falcon · RealSSA-Sim | SI-SNRi: 7.5 -> 7.7 | Improved |
| anneal tau from 0 to 1 instead of fixed tau=1 (ablation) | Falcon · RealSSA-Sim | SI-SNRi: 7.4 -> 7.7 | Improved |
| anneal tau to 2 instead of 1 (ablation) | Falcon · RealSSA-Sim | SI-SNRi: 7.7 -> 7.5 | Hurt |
| train on RealSSA-Sim instead of VoxCeleb2 | Falcon · RealSSA-Real | NISQA: 2.431 -> 3.615 | Improved |
| train on RealSSA-Sim instead of VoxCeleb2 | Falcon · RealSSA-Real | DNSMOS: 2.878 -> 3.276 | Improved |
| train on RealSSA-Sim instead of VoxCeleb2 | five causal architectures · RealSSA-Real | NISQA | Mixed |
| single-pass multi-scale separator instead of iterative Swift-Net-12 | Falcon · RealSSA-Sim | MACs (G): 45.3 -> 10.6 | Improved |
| single-pass separator instead of iterative Swift-Net-12 | Falcon · RealSSA-Sim | GPU latency (ms): 172.1 -> 51.3 | Improved |
Limitations
- Full text unavailable; only the truncated abstract was available
Closest papers by meaning
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.84)
- Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark (CVPR 2024 · similarity 0.84)
- TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation (ICLR 2025 · similarity 0.83)
- RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation (ICLR 2024 · similarity 0.83)
- AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis (NeurIPS 2024 · similarity 0.83)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
