Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning

Phoneme Hallucinator: One-Shot Voice Conversion via Set Expansion

AAAI 2024 · Label-efficient and robust learning › Zero-/few-shot and meta-learning · approach: Zero-shot robustness and transfer · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Phoneme Hallucinator, a conditional set-generative (conditional VAE with set-transformer prior/posterior) model that expands a 3-second target WavLM feature set so kNN-VC becomes intelligible for one-shot voice conversion.

Problem

One-shot voice conversion: neighbor-based VC (kNN-VC) loses intelligibility when only a few seconds of target speech are available, while speaker-embedding methods lack speaker similarity.

Approach

Hallucinate extra 1024-dim WavLM frame representations conditioned on a small target set using a probabilistic set model (de Finetti decomposition, Set Transformer posterior/prior, conditional VAE decoder with modulation), then run kNN regression over the expanded target set.

Key results

• With 3 seconds of target speech, Phoneme Hallucinator achieves WER 5.10% and EER 44.62%, versus FreeVC's 5.40% WER and 35.63% EER and kNN-VC's 45.92% WER.
• Subjective MOS 4.02 (FreeVC 4.05, within error) and SIM 2.98, the highest SIM among compared methods.
• Hallucinator samples 30,000 features in 1.48 s on an RTX 4090, and training takes 6.5 hours.

Measured effects reported in the paper

Surprising findings

• Increasing the number of hallucinated samples improves WER but slightly reduces speaker similarity, which the authors attribute to the neighbor-based pipeline itself rather than the hallucinations.

Limitations

• Vocoder trained only on real representations, not hallucinated ones.
• Trained and evaluated only on English LibriSpeech; cross-lingual results are only qualitative audio samples.
• Speaker similarity slightly decreases as the number of hallucinated samples increases.

Experimental setup

Datasets: LibriSpeech train-clean-100 LibriSpeech dev-clean LibriSpeech test-clean

Benchmarks: LibriSpeech test-clean

Metrics: WER CER EER MOS SIM

Baselines: kNN-VC FreeVC YourTTS VQMIVC

Builds on: kNN-VC Set Transformer De Finetti's theorem POEx HiFi-GAN

Closest papers by meaning

• SelfVC: Voice Conversion With Iterative Refinement using Self Transformations (ICML 2024 · similarity 0.86)
• StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching (AAAI 2025 · similarity 0.86)
• Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement (ICLR 2025 · similarity 0.85)
• Do Not Mimic My Voice : Speaker Identity Unlearning for Zero-Shot Text-to-Speech (ICML 2025 · similarity 0.84)
• RVCBench: Benchmarking Robustness of Voice Cloning Across Modern Audio Generation Models (NeurIPS 2026 · similarity 0.84)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.84)