Research map / Image generation / Autoregressive and latent generation

RVCBench: Benchmarking Robustness of Voice Cloning Across Modern Audio Generation Models

NeurIPS 2026 · Image generation › Autoregressive and latent generation · approach: Neural text-to-speech synthesis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces RVCBench, a benchmark of 18 robustness evaluations (14,370 utterances, 204 speakers) that stress-tests 18 open-source voice cloning models across input, generation, output, and audio-perturbation dimensions.

Problem

Modern voice cloning systems are evaluated mostly under clean, short-form conditions, leaving their robustness to realistic deployment shifts (noisy references, text format shifts, long-form, multilingual, expressive prompts, compression, deepfake detection, adversarial protection) unmeasured.

Approach

A pipeline-level robustness benchmark that reprocesses eight public speech corpora into controlled stress tests spanning four dimensions (input, generation, output, perturbation) and evaluates many open-source zero-shot voice cloning systems under a unified text-plus-reference interface.

Key results

• Evaluates 18 open-source voice cloning models over 18 robustness evaluations, 204 speakers and 14,370 utterance-level items.
• Hallucination-style text prompts consistently increase WER for most models, while scam-style prompts produce smaller changes.
• Compression and narrowband (Phone NB) transmission consistently increase MCD and reduce STOI across all models.
• SafeSpeech and SPEC are reported as the most effective anti-cloning perturbations, while DEMUCS denoising does not fully restore clean-reference quality.

Surprising findings

• Expressive emotion alignment is lower on scam prompts than on benign VCTK prompts despite scam texts carrying stronger affect.
• Simple DEMUCS denoising is generally insufficient to evade anti-cloning protections in current VCL models.

Limitations

• Accent groups are small (some single-speaker), so accent rankings are descriptive only.
• Compression codec and bitrate are not fully crossed, so individual effects cannot be separated.

Experimental setup

Datasets: VCTK LibriTTS LibriSpeech AISHELL-1 Common Voice FR EMIME VoiceBank+DEMAND robocall audio dataset

Benchmarks: RVCBench

Metrics: SIM MOS MCD WER RTF STOI EER minDCF ACC

Baselines: CloneEval AHa-Bench AudioMarkBench EmergentTTS-Eval TTSDS2

Builds on: SafeSpeech Enkidu DEMUCS VoiceBank+DEMAND VCTK

Closest papers by meaning

• De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attacks (ICML 2025 · similarity 0.87)
• Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement (ICLR 2025 · similarity 0.86)
• Advancing Audio-Based Text Generation with Imbalance Preference Optimization (AAAI 2025 · similarity 0.86)
• Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Models (NeurIPS 2025 · similarity 0.85)
• Speech Robust Bench: A Robustness Benchmark For Speech Recognition (ICLR 2025 · similarity 0.85)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.85)