Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDR-TSE, a self-supervised two-phase disentangled representation learning method that extracts speaker-identity embeddings from reference speech and an adaptive modulation Transformer to guide target speech extraction while reducing speaker confusion.
Problem
Target speech extraction with reference speech suffers from speaker confusion, because speaker embeddings entangle semantic and harmful global (emotion, prosody, speaking rate) information; prior remedies need speaker labels or pretrained speaker recognizers.
Approach
A VAE-based reference encoder separates global and semantic latents (with instance norm, mutual information minimization via vCLUB, and reconstruction/KL losses); a channel-attention global disentanglement network then selects speaker-identity channels using a contrastive similarity loss; and adaptive modulation layer norm in the Transformer injects these speaker embeddings as conditions without d
Key results
- On WSJ0-2mix-extr, SDR-TSE reaches 19.9 SI-SNRi, 20.2 SDRi, 3.85 PESQ and 6.89 r_scr, outperforming SpEx+ (15.7 SI-SNRi).
- On WSJ0-2mix, SDR-TSE reaches 19.6 SI-SNRi and 6.67 r_scr versus X-SepFormer (S_sc) at 19.1 and 8.56.
- Ablation: removing MI minimization drops SI-SNRi from 19.9 to 17.3; removing L_REC drops it to 16.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add disentangled speaker identity embedding z_s instead of global information z_g | SDR-TSE speech extraction network · WSJ0-2mix-extr | SI-SNRi: 19.3 -> 19.9 | Helped |
| add disentangled speaker identity embedding z_s instead of semantic information z_c | SDR-TSE speech extraction network · WSJ0-2mix-extr | SI-SNRi: 15.6 -> 19.9 | Helped |
| add instance normalization in E_c (ablation) | SDR-TSE · WSJ0-2mix-extr | SI-SNRi: 18.2 -> 19.9 | Helped |
| add mutual information minimization (ablation) | SDR-TSE · WSJ0-2mix-extr | SI-SNRi: 17.3 -> 19.9 | Helped |
| add L_SIM (ablation) | SDR-TSE · WSJ0-2mix-extr | SI-SNRi: 18.4 -> 19.9 | Helped |
| add L_REC (ablation) | SDR-TSE · WSJ0-2mix-extr | SI-SNRi: 16.2 -> 19.9 | Helped |
| add L_KL (ablation) | SDR-TSE · WSJ0-2mix-extr | SI-SNRi: 18.8 -> 19.9 | Helped |
| add 2-D positional encoding (ablation) | SDR-TSE · WSJ0-2mix-extr | SI-SNRi: 19.4 -> 19.9 | Helped |
| use AMLN rather than summation to fuse speaker embeddings and acoustic representations | SDR-TSE speech extraction network · WSJ0-2mix-extr | SI-SNRi: 18.7 -> 19.9 | Helped |
| use AMLN rather than concatenation to fuse speaker embeddings and acoustic representations | SDR-TSE speech extraction network · WSJ0-2mix-extr | SI-SNRi: 19.0 -> 19.9 | Helped |
| use AMLN rather than Gated Conv for modulation | SDR-TSE speech extraction network · WSJ0-2mix-extr | SI-SNRi: 19.3 -> 19.9 | Helped |
| use AMLN rather than ConSM for modulation | SDR-TSE speech extraction network · WSJ0-2mix-extr | SI-SNRi: 19.2 -> 19.9 | Helped |
Surprising findings
- Using only the disentangled speaker-identity embedding z_s outperforms using the global embedding z_g alone, and semantic-only guidance performs worst (12.14 r_scr).
Limitations
- Speaker confusion metric and evaluation limited to two-speaker WSJ0 mixtures.
- Relies on a multi-component pipeline with many loss weights tuned by grid search (model has 45M parameters).
Experimental setup
Datasets: WSJ0-2mix WSJ0-2mix-extr
Benchmarks: WSJ0-2mix-extr WSJ0-2mix
Metrics: SI-SNRi SDRi PESQ r_scr (chunk-wise speaker confusion ratio)
Baselines: TseNet SpEx SpEx+ DPRNN-Spe-IRA X-SepFormer Gated Conv ConSM
Builds on: Sepformer vCLUB AdaIN X-SepFormer Conv-TasNet
Closest papers by meaning
- SelfVC: Voice Conversion With Iterative Refinement using Self Transformations (ICML 2024 · similarity 0.86)
- Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations (AAAI 2024 · similarity 0.85)
- Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation (NeurIPS 2024 · similarity 0.84)
- CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing (NeurIPS 2024 · similarity 0.84)
- DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion (AAAI 2024 · similarity 0.84)
- Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.