Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a negation-based speaker representation (subtracting content features from EnCodec audio features) with multi-stream Transformers and attention pooling, plus SALN fusion, for zero-shot multi-speaker TTS built on FastSpeech.
Problem
Zero-shot multi-speaker TTS suffers from poor speaker disentanglement and content leakage into speaker embeddings, hurting speaker similarity and robustness on unseen (out-of-domain) speakers.
Approach
Derive content-information-free speaker embeddings by subtracting content features (instance-normalization content extractor) from a full EnCodec audio representation, process them with parallel multi-stream Transformers whose hypotheses are merged by attention pooling, and fuse the speaker embedding with text via adaptive layer normalization (SALN) in encoder and decoder.
Key results
- Proposed model achieves MCD 5.59 / MOS 3.86 / WER 15.37% on seen speakers and MCD 8.57 / MOS 3.84 / WER 16.14% on unseen speakers, beating Mellotron and Meta-StyleSpeech on all three metrics.
- Removing negation raises MCD from 7.08 to 7.44 and drops MOS from 3.85 to 3.30, and WER rises from 15.53% to 16.68%.
- Attention pooling over streams improves MOS over concatenation (3.85 vs 3.51) and single-stream gives MOS 3.49.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add negated feature generation to remove content information from the full audio representation (ablation) | Ours · Seen and unseen speakers (averaged) | MCD: 7.44 -> 7.08 | Helped |
| add negated feature generation to remove content information from the full audio representation (ablation) | Ours · Seen and unseen speakers (averaged) | MOS: 3.30 ± 0.23 -> 3.85 ± 0.12 | Helped |
| add negated feature generation to remove content information from the full audio representation (ablation) | Ours · Seen and unseen speakers (averaged) | WER: 16.68% -> 15.53% | Helped |
| replace Wav2Vec Layer 1 with instance normalization as the content extractor (ablation) | Ours · Seen and unseen speakers (averaged) | MCD: 7.51 -> 7.08 | Helped |
| replace Wav2Vec Layer 1 with instance normalization as the content extractor (ablation) | Ours · Seen and unseen speakers (averaged) | MOS: 3.43 ± 0.15 -> 3.85 ± 0.12 | Helped |
| replace Wav2Vec Layer 1 with instance normalization as the content extractor (ablation) | Ours · Seen and unseen speakers (averaged) | WER: 16.42% -> 15.56% | Helped |
| replace Wav2Vec Layer 7 with instance normalization as the content extractor (ablation) | Ours · Seen and unseen speakers (averaged) | MCD: 7.80 -> 7.08 | Helped |
| replace Wav2Vec Layer 7 with instance normalization as the content extractor (ablation) | Ours · Seen and unseen speakers (averaged) | MOS: 3.12 ± 0.19 -> 3.85 ± 0.12 | Helped |
| replace Wav2Vec Layer 7 with instance normalization as the content extractor (ablation) | Ours · Seen and unseen speakers (averaged) | WER: 16.79% -> 15.56% | Helped |
| replace Wav2Vec Layer 12 with instance normalization as the content extractor (ablation) | Ours · Seen and unseen speakers (averaged) | MCD: 7.77 -> 7.08 | Helped |
| replace Wav2Vec Layer 12 with instance normalization as the content extractor (ablation) | Ours · Seen and unseen speakers (averaged) | MOS: 3.02 ± 0.21 -> 3.85 ± 0.12 | Helped |
| replace Wav2Vec Layer 12 with instance normalization as the content extractor (ablation) | Ours · Seen and unseen speakers (averaged) | WER: 16.65% -> 15.56% | Helped |
Surprising findings
- Best performance came with the smallest number of heads and blocks, which the authors attribute to the short average utterance duration of the training data.
- Wav2Vec layer 1 outperformed layers 7 and 12 as a content extractor, though instance normalization still did better.
Limitations
- Unseen-speaker MCD is higher than seen-speaker MCD, and the proposed model is not best on all unseen MCD configurations
- Evaluation limited to LibriTTS train-clean-100 and two baselines
Experimental setup
Datasets: LibriTTS train-clean-100
Benchmarks: LibriTTS train-clean-100 seen and unseen speakers
Metrics: MCD MOS WER ABX preference
Baselines: Mellotron Meta-StyleSpeech Wav2Vec content extractor variants
Builds on: FastSpeech EnCodec Style-Adaptive Layer Normalization multi-stream Transformer HiFi-GAN
Closest papers by meaning
- Do Not Mimic My Voice : Speaker Identity Unlearning for Zero-Shot Text-to-Speech (ICML 2025 · similarity 0.90)
- Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis (ICLR 2024 · similarity 0.89)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.88)
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (ICML 2024 · similarity 0.88)
- CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR 2024 · similarity 0.87)
- StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.