Research map / Adversarial robustness and security / Attacks on text and graphs
De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attacks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically evaluates protective perturbations against zero-shot voice cloning under purification attacks, and proposes PhonePuRe, a two-stage diffusion purification (DiffWave purification plus phoneme-guided score-based refinement) that better restores speaker identity in cloned speech.
Problem
Protective adversarial perturbations that prevent zero-shot voice cloning may be neutralized by adversarial purification, and existing purifiers designed for classification distort speaker-relevant features in voice cloning embedding spaces.
Approach
A two-stage purification-refinement framework: an unconditional DiffWave purifier removes perturbations, then a phoneme-guided score-based (NCSN++, OUSDE) refiner trained on clean vs. purified clean pairs maps purified speech toward the clean distribution.
Key results
- Under AntiFake protection, PhonePuRe reaches dSVA of 76.2%, surpassing existing purifiers by at least 31.1%.
- Averaged over protections and VC models, PhonePuRe attains xSVA 0.711 and dSVA 0.818, above AudioPure (0.597 / 0.646).
- Even at EOT size 15 under adaptive BPDA attacks, PhonePuRe keeps dSVA above 0.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PhonePuRe instead of the strongest baseline, AudioPure | Voice cloning from protected speech · LibriSpeech test-clean evaluation set, Total | xSVA: 0.597 -> 0.711 | Helped |
| Use PhonePuRe instead of the strongest baseline, AudioPure | Voice cloning from protected speech · LibriSpeech test-clean evaluation set, Total | dSVA: 0.647 -> 0.818 | Helped |
| Add the purification stage (ablation) | PhonePuRe · LibriSpeech test-clean evaluation set, Total | xSVA: 0.350 -> 0.711 | Helped |
| Add the refinement stage (ablation) | PhonePuRe · LibriSpeech test-clean evaluation set, Total | xSVA: 0.597 -> 0.711 | Helped |
| Add phoneme guidance (ablation) | PhonePuRe refinement stage · LibriSpeech test-clean evaluation set, Total | dSVA: 0.786 -> 0.818 | Helped |
| Add data augmentation during refinement training (ablation) | PhonePuRe · LibriSpeech test-clean evaluation set, Total | xSVA: 0.682 -> 0.711 | Helped |
| Vary the purification-stage diffusion steps (ablation) | PhonePuRe purification stage · AttackVC | SVA | Mixed |
| Vary the refinement-stage sampling steps (ablation) | PhonePuRe refinement stage · Evaluation set | SVA | No clear effect |
Surprising findings
- Existing purifiers can raise xSVA of protected speech to an average of about 60% (AudioPure), showing protective perturbations are largely neutralized.
- Transformation-based purifiers (WaveGuard, SpeakerGuard) and low mel-bin diffusion purifiers perform poorly, often below 0.25 average SVA.
Limitations
- Evaluation limited to three protection methods and six VC models, with white-box assumptions for some protections.
- Adaptive protection via BPDA/adjoint is approximate due to deep diffusion computation graphs.
- Purification introduces some muffling and artifacts, lowering perceived naturalness relative to clean speech.
Experimental setup
Datasets: LibriSpeech DEMAND
Benchmarks: LibriSpeech test-clean (25 speakers, 5 sentences each)
Metrics: xSVA dSVA NISQA MOS human speaker similarity (four-level scale)
Baselines: WaveGuard SpeakerGuard AudioPure WavePurifier DualPure
Builds on: AntiFake DiffWave AudioPure NCSN++ Montreal Forced Aligner
Closest papers by meaning
- RVCBench: Benchmarking Robustness of Voice Cloning Across Modern Audio Generation Models (NeurIPS 2026 · similarity 0.87)
- E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis (NeurIPS 2025 · similarity 0.87)
- Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation (CVPR 2025 · similarity 0.86)
- ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio–Language Models (NeurIPS 2025 · similarity 0.84)
- AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models (ICLR 2025 · similarity 0.84)
- DiffHammer: Rethinking the Robustness of Diffusion-Based Adversarial Purification (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.