Research map / Label-efficient and robust learning / Recognition and long-tail
Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes target speaker extraction conditioned on noisy enrollments by contrasting Positive (target talking) and Negative (target silent) segments with a Siamese TF-GridNet encoder, attention fusion, and two-stage distillation training.
Problem
Target speaker extraction usually needs clean audio enrollments, which are often unavailable; prior noisy-enrollment methods either need only-target segments or assume the user was in the recording, limiting arbitrary mixtures.
Approach
Encode the target speaker from a noisy Positive Enrollment paired with a Negative Enrollment where the target is silent, exploiting the temporal non-synchrony of speakers; a Siamese TF-GridNet encoder with attention-based Encoder Fusion compares the two, is distilled from a frozen clean-speech encoder in stage one, and the extraction branch is trained in stage two with cross-attention fusion.
Key results
- Monaural model achieves SI-SNRi of 8.85 dB with 2 speakers in mixture and 2 speakers in enrollment, versus 6.67 dB for TCE in the same setting.
- Two-stage training reaches 3 dB SNR on validation with 240k optimization steps versus 600k for end-to-end training (60% fewer steps).
- Real-world MOS of 3.35 on a 1-4 clarity scale, outperforming baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use two-stage training instead of end-to-end training (ablation) | Ours (Monaural) · validation set | optimization steps to reach 3 dB SNR: 600k -> 240k optimization steps; 60% fewer optimization steps | Helped |
| replace cross-attention fusion with FiLM fusion (ablation) | Ours (Monaural) · LibriSpeech test-clean, mono target extraction | SI-SNRi: 8.85$\pm$3.67 -> 6.70$\pm$3.43 (2-speaker mixture, 2-speaker enrollment) | Hurt |
| use pooling size 20 instead of 40 (ablation) | Ours (Monaural) · LibriSpeech test-clean, mono, 2-speaker mixture, 2-speaker enrollment | SI-SNRi: 8.85$\pm$3.67 -> 8.10$\pm$3.90 | Hurt |
| use pooling size 80 instead of 40 (ablation) | Ours (Monaural) · LibriSpeech test-clean, mono, 2-speaker mixture, 2-speaker enrollment | SI-SNRi: 8.85$\pm$3.67 -> 8.59$\pm$3.77 | Hurt |
| increase the number of Positive Interferers from 1 to 3 (ablation) | Ours (Monaural) · LibriSpeech test-clean, 2-speaker mixture | SI-SNRi: 8.98$\pm$3.66 -> 8.79$\pm$3.76 | Hurt |
| increase the number of Negative Interferers from 1 to 3 (ablation) | Ours (Monaural) · LibriSpeech test-clean, 2-speaker mixture | SI-SNRi: 8.69$\pm$4.06 -> 7.43$\pm$5.39 | Hurt |
| increase positive enrollment duration from 1 sec to 3 sec (ablation) | Ours (Monaural) · LibriSpeech test-clean, negative enrollment 1 sec | SI-SNRi: 7.07$\pm$4.45 -> 8.16$\pm$3.94 | Helped |
| increase negative enrollment duration from 1 sec to 3 sec while positive enrollment is 1 sec (ablation) | Ours (Monaural) · LibriSpeech test-clean, positive enrollment 1 sec | SI-SNRi: 7.07$\pm$4.45 -> 7.16$\pm$4.38 | Mixed |
| use user-provided inaccurate enrollment labels instead of ground-truth labels (ablation) | Ours (Monaural) · 10 manually generated samples | SI-SDRi | No clear effect |
| add Negative Enrollment to the binaural enrollment strategy | Ours (Binaural) · LibriSpeech test-clean, binaural target extraction | SNRi | Helped |
Surprising findings
- Model degrades only about 0.2 dB SI-SNRi as the number of Positive Interferers grows from 1 to 3, but over 1 dB as Negative Interferers increase.
- Smaller embedding pooling size (20) performed worse than pooling size 40 despite retaining more detail.
Limitations
- Two-stage training may cap encoder performance at the frozen distillation teacher's quality.
- Extracted audio contains noticeable artifacts, reflected in low DNSMOS scores.
- Does not outperform SOTA models trained specifically on clean enrollments.
Experimental setup
Datasets: LibriSpeech WHAM! CIPIC RRBRIR ASH-Listening-Set CATTRIR MSDWild VoxConverse
Benchmarks: LibriSpeech test-clean simulated mixtures
Metrics: SNRi SI-SNRi PESQ STOI DNSMOS WER
Baselines: TCE USEF-TFGridnet NMF LookOnceToHear
Builds on: TF-GridNet LookOnceToHear TCE USEF-TSE
Closest papers by meaning
- Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations (AAAI 2024 · similarity 0.85)
- Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction (AAAI 2024 · similarity 0.84)
- Do Not Mimic My Voice : Speaker Identity Unlearning for Zero-Shot Text-to-Speech (ICML 2025 · similarity 0.84)
- Weakly-supervised Audio Separation via Bi-modal Semantic Similarity (ICLR 2024 · similarity 0.82)
- Rethinking Training Targets, Architectures and Data Quality for Universal Speech Enhancement (NeurIPS 2026 · similarity 0.82)
- StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.