Research map / Detection and segmentation / CNN features
Proactive Detection of Voice Cloning with Localized Watermarking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AudioSeal, a jointly trained generator/detector audio watermark that performs sample-level localized detection of AI-generated speech with a single-pass detector and a loudness-based perceptual loss.
Problem
Detecting AI-generated speech (voice cloning) and localizing watermarked segments within longer audio, where prior multi-bit watermarks are not adapted for detection, are not localized, and have slow brute-force detection.
Approach
Jointly train a watermark generator and a sample-level detector with masked-augmentation localization loss, differentiable robustness augmentations, and a time-frequency loudness loss inspired by auditory masking; detector runs once and outputs per-sample logits, with optional multi-bit attribution heads.
Key results
- Average AUC over 15 audio edits of 0.97 for AudioSeal vs 0.84 for WavMark.
- IoU of 0.99 for 1 second of watermarked speech in 10 s audio vs 0.35 for WavMark.
- Detection is 485x faster than WavMark on non-watermarked audio and generation is 14x faster on watermarked audio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AudioSeal instead of WavMark for watermarking | Audio watermark detection on edited VoxPopuli validation audio · VoxPopuli validation, edited audio average | AUC: 0.84 -> 0.97 | Helped |
| use AudioSeal instead of WavMark for watermarking | Localization of a short watermarked segment · 1 second watermark in 10 second audio | IoU: 0.35 -> 0.99 | Helped |
| use AudioSeal instead of Wavmark for watermark detection | Detection of unwatermarked audio · Watermark detection, no watermark | runtime: 485$\times$ faster | Helped |
| use AudioSeal instead of WavMark for watermarking | Speech watermark perceptual quality · Speech quality | PESQ: 4.302 -> 4.470 | Helped |
| use AudioSeal instead of WavMark for watermarking | Speech watermark perceptual quality · Speech quality | SI-SNR: 38.25 -> 26.00 | Hurt |
| use AudioSeal instead of the Voicebox binary classifier | Detection of Voicebox-generated versus re-synthesized audio · Voicebox vs re-synthesized, 30% masked | accuracy: 0.704 -> 1.0 | Helped |
| replace EnCodec architecture with DPRNN (ablation) | AudioSeal · 1 second watermark in 10 second audio | IoU: 0.802 -> 0.796 | No clear effect |
| use AudioSeal instead of WavMark for attribution | Attribution among model versions · Attribution, 1 model version | attribution accuracy (%): 58.4 -> 68.2 | Helped |
| use AudioSeal instead of WavMark for attribution | Attribution among model versions · Attribution, 10^3 model versions | attribution accuracy (%): 56.6 -> 59.3 | Helped |
Surprising findings
- WavMark's detection bit-level scores on genuine audio are centered around 0.38 rather than 0.5, making theoretical FPR guarantees invalid in practice.
- Watermark-removal via white-box adversarial attack increases detection error by about 80% while keeping PESQ above 4, so detector weights must be kept confidential.
Limitations
- Attribution has higher false attribution rate (FAR) than WavMark, rising to 11.84% at N=10^4.
- Highpass filtering above training range (500Hz) yields much lower accuracy than WavMark (0.61 vs 1.00 AUC).
- Detector is vulnerable to white-box and black-box adversarial removal if its weights or training-based classifier leak.
Experimental setup
Datasets: VoxPopuli LibriSpeech ASVspoof FakeAVCeleb Expresso AudioSet
Benchmarks: VoxPopuli validation edits ASVspoof FakeAVCeleb AudioSet LibriSpeech Voicebox samples
Metrics: accuracy TPR FPR AUC IoU sample-level accuracy SI-SNR PESQ STOI ViSQOL
Baselines: WavMark Voicebox binary classifier
Builds on: EnCodec WavMark HiFi-GAN
Closest papers by meaning
- AudioMarkBench: Benchmarking Robustness of Audio Watermarking (NeurIPS 2024 · similarity 0.88)
- XAttnMark: Learning Robust Audio Watermarking with Cross-Attention (ICML 2025 · similarity 0.86)
- Speech Watermarking with Discrete Intermediate Representations (AAAI 2025 · similarity 0.85)
- Asymmetric Phase Coding Audio Watermarking (NeurIPS 2026 · similarity 0.85)
- Robust Distortion-Free Watermark for Autoregressive Audio Generation Models (NeurIPS 2025 · similarity 0.84)
- RVCBench: Benchmarking Robustness of Voice Cloning Across Modern Audio Generation Models (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.