AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Wendi Sang, Kai Li, Yifan Li, Jianqiang Huang, Xiaolin Hu

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual source localization · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Targets streaming audio-visual target speaker extraction for real-world acoustic scenes with lightweight, edge-deployable causal models.

Problem

Current causal streaming audio-visual target speaker extraction (AVTSE) systems are evaluated on benchmarks with exactly two speakers that ignore near/far-field interference, reverberation, and sudden noise, and high-performing causal architectures are too heavy for edge deployment.

Approach

Abstract is truncated; the paper pairs more realistic multi-speaker, reverberant, noisy evaluation conditions with a lightweight causal AVTSE architecture for edge deployment. Specific architectural details are not stated in the available text.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase separator downsampling stages from one to three (ablation)Falcon · RealSSA-SimSI-SNRi: 7.1 -> 7.7Improved
add encoder-space speaker contrastive loss (ablation)Falcon · RealSSA-SimSI-SNRi: 7.3 -> 7.7Improved
add encoder-space speaker contrastive lossSwiftNet-12 · RealSSA-SimSI-SNRi: 7.3 -> 7.5Improved
add encoder-space speaker contrastive lossDolphin · RealSSA-SimSI-SNRi: 6.7 -> 7.0Improved
anneal contrastive temperature tau from 0 to 1 instead of fixed tau=0 (ablation)Falcon · RealSSA-SimSI-SNRi: 7.5 -> 7.7Improved
anneal tau from 0 to 1 instead of fixed tau=1 (ablation)Falcon · RealSSA-SimSI-SNRi: 7.4 -> 7.7Improved
anneal tau to 2 instead of 1 (ablation)Falcon · RealSSA-SimSI-SNRi: 7.7 -> 7.5Hurt
train on RealSSA-Sim instead of VoxCeleb2Falcon · RealSSA-RealNISQA: 2.431 -> 3.615Improved
train on RealSSA-Sim instead of VoxCeleb2Falcon · RealSSA-RealDNSMOS: 2.878 -> 3.276Improved
train on RealSSA-Sim instead of VoxCeleb2five causal architectures · RealSSA-RealNISQAMixed
single-pass multi-scale separator instead of iterative Swift-Net-12Falcon · RealSSA-SimMACs (G): 45.3 -> 10.6Improved
single-pass separator instead of iterative Swift-Net-12Falcon · RealSSA-SimGPU latency (ms): 172.1 -> 51.3Improved

Limitations

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy