Research map / Large language models / Language, safety and interpretability

E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Speech and audio language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes E2E-VGuard, a proactive adversarial-perturbation defense that protects audio against end-to-end (ASR-transcribed) fine-tuning and zero-shot voice cloning of LLM-based TTS by combining timbre encoder-ensemble feature loss, ASR-targeted adversarial loss, and a psychoacoustic imperceptibility term.

Problem

Voice-cloning fraud via production LLM-based TTS and end-to-end pipelines where transcripts are produced automatically by ASR, which existing defenses (AEs and unlearnable examples assuming manual transcripts) do not handle.

Approach

Jointly perturb audio to (1) maximize feature distance from an ensemble of timbre encoders plus MFCC, (2) force a target ASR to output a chosen text so the TTS learns wrong text-pronunciation alignment, and (3) keep the perturbation under a psychoacoustic frequency-masking threshold with an L2 penalty.

Key results

• On fine-tuning TTS models, E2E-VGuard (T) raises WER by an average of 19.775% over the best baseline while reducing SIM by 0.043 (UT).
• On 7 zero-shot industrial/LLM TTS models, E2E-VGuard (UT) averages 21.603% WER versus AntiFake's 4.932%.
• On commercial APIs, average SIM drops from 0.689 to 0.203 with protection.

Measured effects reported in the paper

Surprising findings

• Removing the psychoacoustic and L2 terms yields stronger protection but much lower SNR (12.942), showing a protection-imperceptibility trade-off.
• Protection still holds with clean ground-truth text in fine-tuning, though timbre protection weakens (SIM 0.278 UT).

Limitations

• Pronunciation protection targets a specific ASR (Wav2vec2) and relies on transfer to other systems.
• Protection time per sample is about 100 seconds, which may be costly at scale.
• Imperceptibility is not guaranteed; the paper argues usability rather than indistinguishability.

Experimental setup

Datasets: LibriTTS CMU ARCTIC THCHS30

Benchmarks: LibriTTS CMU ARCTIC THCHS30

Metrics: WER SIM SNR PESQ MOS

Baselines: AttackVC AntiFake POP POP+ESP SafeSpeech

Builds on: AntiFake V-Cloak SafeSpeech PGD

Closest papers by meaning

• De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attacks (ICML 2025 · similarity 0.87)
• RVCBench: Benchmarking Robustness of Voice Cloning Across Modern Audio Generation Models (NeurIPS 2026 · similarity 0.85)
• ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio–Language Models (NeurIPS 2025 · similarity 0.84)
• AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models (ICLR 2025 · similarity 0.84)
• Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Models (NeurIPS 2025 · similarity 0.84)
• Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (ICLR 2024 · similarity 0.83)