Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
ELF: Encoding Speaker-Specific Latent Speech Feature for Speech Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ELF, a VAE-based speech feature encoder whose k-means-clustered latent codebook per speaker is fused via attention into a VITS-style TTS model, enabling speaker modeling without per-speaker training.
Problem
Zero-shot and fine-tuning speaker adaptation for TTS fail to capture a speaker's overall, content-dependent speech characteristics (timbre, prosody, pronunciation) as well as multi-speaker trained models, and zero-shot prompting copies the prompt rather than adapting to new content.
Approach
Encode each speaker's mel-spectrograms with a variational autoencoder (SFEN), cluster the latent means into a discrete per-speaker codebook of 512 centroids, and fuse codebook vectors into the TTS text encoder via multi-head attention so that attention-weighted combinations sample a continuous latent space; blending speakers is done by mixing their attention-fused features with weights.
Key results
- ELF with all audio achieved SMOS 3.85 on unseen speakers, above VITS multi-speaker (seen speakers) at 3.73, and ELF with one audio sample got SECS 0.848 vs YourTTS 0.817.
- ELF with 20 audio samples reached CER 1.26 versus 1.80 for multi-speaker VITS, indicating higher intelligibility.
- Speech feature-to-speech (codebook only, no text condition beyond alignment) achieved MOS 3.91 and CER 1.36 against ground truth 3.96 and 1.29.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ELF with all available target-speaker audio instead of VITS multi-speaker training | ELF · LibriTTS and VCTK test | SMOS: 3.73 ($\pm$0.09) -> \textbf{3.85} ($\pm$0.09) | Helped |
| Use ELF with one reference audio instead of YourTTS | ELF · LibriTTS and VCTK test | SMOS: 3.47 ($\pm$0.11) -> 3.62 ($\pm$0.10) | Helped |
| Increase target-speaker audio used for the codebook from 1 to 20 samples (ablation) | ELF · LibriTTS and VCTK test | SMOS: 3.62 ($\pm$0.10) -> 3.76 ($\pm$0.10) | Helped |
| Increase target-speaker audio used for the codebook from 20 samples to all available audio (ablation) | ELF · LibriTTS and VCTK test | SMOS: 3.76 ($\pm$0.10) -> \textbf{3.85} ($\pm$0.09) | Helped |
| Increase target-speaker audio used for the codebook from 1 to 20 samples (ablation) | ELF · LibriTTS and VCTK test | CER: 2.35 -> 1.26 | Helped |
| Blend speech features from multiple speakers (ablation) | ELF · Speaker blending, 8 speakers | MOS: ELF (1 speaker): 3.89 (0.10); ELF (8 speakers): 3.96 (0.09) | No clear effect |
| Increase blending speaker count from 1 to 2 (ablation) | ELF · Speaker blending, 2 speakers | MOS: 3.89 (0.10) -> 3.83 (0.10) | No clear effect |
| Increase blending speaker count from 1 to 4 (ablation) | ELF · Speaker blending, 4 speakers | MOS: 3.89 (0.10) -> 3.86 (0.09) | No clear effect |
| Increase blending speaker count from 1 to 8 (ablation) | ELF · Speaker blending, 8 speakers | CER: 1.19 -> 1.03 | Helped |
| Use ELF rather than YourTTS for cross-lingual synthesis | ELF · Cross-lingual TTS | MOS: 3.29 (0.13) -> \textbf{3.79} (0.12) | Helped |
| Use ELF rather than YourTTS for cross-lingual synthesis | ELF · Cross-lingual TTS | CER: 7.91 -> 2.84 | Helped |
| Synthesize speech from the encoded codebook features with text conditioning | Speech Feature-to-Speech · Speech feature-to-speech | MOS: Ground Truth: 3.96 (0.09); Speech Feature-to-Speech: 3.91 (0.10) | No clear effect |
Surprising findings
- Zero-shot YourTTS showed a biased blend at 5:5 ratio, suggesting speaker similarity is not tied to distance in its embedding space, while ELF showed smooth similarity change.
- The codebook alone, without text input, was sufficient to synthesize speech close to ground truth quality.
Limitations
- Zero-shot (1-audio) variant is weaker than multi-speaker VITS in SECS and speaker verification EER (16.9 vs 10.8).
- Authors note the VAE aggregated posterior does not perfectly match the Gaussian prior, so the learned space is only approximately Gaussian.
- Evaluation uses only 500 samples and crowd-sourced raters from North America; the SMOS comparison is limited to two reference models.
Experimental setup
Datasets: LibriTTS VCTK Korean dataset (unnamed public set) TAU Urban Acoustic Scenes 2021 Mobile (noise robustness)
Benchmarks: LibriTTS VCTK
Metrics: SMOS MOS CER SECS EER minDCF
Baselines: VITS multi-speaker YourTTS
Builds on: VITS HiFi-GAN YourTTS
Closest papers by meaning
- Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations (AAAI 2024 · similarity 0.86)
- Metis: A Foundation Speech Generation Model with Masked Generative Pre-training (NeurIPS 2025 · similarity 0.85)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.84)
- SelfVC: Voice Conversion With Iterative Refinement using Self Transformations (ICML 2024 · similarity 0.84)
- Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation (NeurIPS 2024 · similarity 0.83)
- Audio Large Language Models Can Be Descriptive Speech Quality Evaluators (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.