Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning

Do Not Mimic My Voice : Speaker Identity Unlearning for Zero-Shot Text-to-Speech

ICML 2025 · Label-efficient and robust learning › Zero-/few-shot and meta-learning · approach: Robust fine-tuning of zero-shot models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes speaker identity unlearning for zero-shot TTS (VoiceBox) via Teacher-Guided Unlearning, which trains the model to produce random teacher-generated voices for forget speakers, plus a new spk-ZRF randomness metric.

Problem

Zero-shot TTS models can clone unseen voices from a 3-second prompt, so an individual who opts out cannot be prevented from being replicated; existing machine unlearning methods do not address removing the ability to mimic specific speaker voices.

Approach

Guide the model, when given a forget speaker's audio prompt, to synthesize the text with the teacher's prompt-free output (random voice style, obtained from Gaussian-noise initialization), combined with CFM loss on remain speakers; a spk-ZRF metric measures speaker-identity randomness via speaker embeddings and JSD.

Key results

• TGU keeps remain-speaker SIM at 0.631 vs 0.649 for the original VoiceBox while reducing forget-speaker SIM to 0.169 (original 0.708).
• TGU reaches spk-ZRF-F of 0.871 vs 0.846 for the original model, the highest among compared methods on the forget set.
• NG and KL collapse with high WER (6.1 and 47.2 on forget set), showing that low SIM alone is misleading.

Measured effects reported in the paper

Surprising findings

• Exact unlearning and fine-tuning on remain speakers only leave forget-speaker mimicry essentially unchanged (SIM-F 0.687 and 0.675), showing simply excluding speakers is insufficient.
• Low forget-set SIM for NG and KL coincides with high WER, meaning they output unintelligible noise rather than a different voice.

Limitations

• SGU and TGU both reduce remain-speaker similarity relative to the original (TGU by 2.8%, SGU by 21%), and diversity (FSD) is reduced with TGU.
• Evaluation uses only a single base model (VoiceBox) and a LibriHeavy/LibriSpeech setup with 10 forget speakers.
• Human evaluation involved 25 participants and small effect sizes on CMOS-R.

Experimental setup

Datasets: LibriHeavy LibriTTS LibriSpeech test-clean LibriSpeech test-other

Benchmarks: LibriSpeech test-clean LibriTTS LibriSpeech test-other

Metrics: WER SIM spk-ZRF CMOS SMOS FSD

Baselines: Exact Unlearning Fine Tuning Negative Gradient Selective KL divergence SGU

Builds on: VoiceBox Zero Retrain Forgetting negative gradient unlearning selective KL unlearning

Closest papers by meaning

• Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations (AAAI 2024 · similarity 0.90)
• Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis (ICLR 2024 · similarity 0.88)
• Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement (ICLR 2025 · similarity 0.88)
• StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching (AAAI 2025 · similarity 0.87)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.86)
• DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis (ICML 2025 · similarity 0.85)