Research map / Adversarial robustness and security / Attacks on text and graphs
AudioMarkBench: Benchmarking Robustness of Audio Watermarking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AudioMarkBench, the first systematic benchmark of audio watermark robustness against removal and forgery under no-box, black-box, and white-box perturbations, with a new Common Voice-derived dataset (AudioMarkData).
Problem
Audio watermarks for AI-generated speech are assumed robust, but their resistance to common edits and adversarial watermark removal or forgery has not been systematically benchmarked, nor has fairness across speaker attributes.
Approach
Builds a benchmark that adapts image-domain attacks (HopSkipJump, Square attack, white-box gradient methods) and common audio edits to audio watermark detectors, evaluates them on a new demographically balanced dataset sampled from Common Voice, and measures FNR/FPR with SNR and ViSQOL quality constraints.
Key results
- Under no-box perturbations, EnCodec, SoundStream, and Opus achieve high FNR while ViSQOL stays at or above 3, showing watermark removal without quality loss.
- White-box perturbations at SNR 20 push FNR to 1.0 for all four watermarking methods on AudioMarkData, while preserving audio quality.
- Watermarked female-voice audio shows significantly higher FNR than male under Gaussian noise (two-tailed t-test p≈2.4e-6), indicating a fairness gap.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply EnCodeC, SoundStream, or Opus watermark-removal perturbations at settings preserving ViSQOL no smaller than 3 | AudioSeal, AudioSeal-B, Timbre, and WavMark · AudioMarkData and LibriSpeech | FNR | Hurt |
| Apply waveform-domain HSJA watermark-removal attack | Audio watermarking methods · AudioMarkData and LibriSpeech | SNR/ViSQOL | No clear effect |
| Optimize HSJA watermark-removal perturbations in the spectrogram domain for more iterations | AudioSeal-B, Timbre, and WavMark · AudioMarkData and LibriSpeech | SNR/ViSQOL: WavMark achieving SNR/ViSQOL of 40/4.5; AudioSeal-B and Timbre reaching approximately 30/4 | Helped |
| Use spectrogram rather than waveform domain for HSJA watermark removal | AudioSeal · AudioMarkData and LibriSpeech | SNR/ViSQOL: below 10/3 | Hurt |
| Apply sufficient-query black-box HSJA watermark-removal attacks | Audio watermarking methods · AudioMarkData and LibriSpeech | FNR | Hurt |
| Apply existing black-box watermark-forgery attacks | Audio watermarking methods · AudioMarkData and LibriSpeech | FPR | No clear effect |
| Apply white-box watermark-removal perturbations at SNR 20 | AudioSeal, AudioSeal-B, Timbre, and WavMark · AudioMarkData and LibriSpeech | FNR: FNRs reach 1 for all watermarking methods when the SNR of the perturbations is 20 | Hurt |
| Apply white-box watermark-forgery perturbations that preserve audio quality | AudioSeal, AudioSeal-B, Timbre, and WavMark · AudioMarkData and LibriSpeech | FPR | Hurt |
| Use AudioSeal instead of the other evaluated watermarking methods | Audio watermarking methods · No-box perturbations | FNR/FPR | Helped |
| Use WavMark instead of the other evaluated watermarking methods | Audio watermarking methods · No-box perturbations | FNR/FPR | Hurt |
| Apply watermark-removal Gaussian noise to female-group audio rather than male-group audio | AudioSeal, AudioSeal-B, Timbre, and WavMark · AudioMarkData, biological-sex groups | FNR: p-value≈2.4×10^{-6}<α=0.05 | Hurt |
| Apply watermark-forgery EnCodec to female-group unwatermarked audio rather than male-group audio | AudioSeal · AudioMarkData, biological-sex groups | FPR | Hurt |
Surprising findings
- Watermarks are robust to common no-box edits like time stretch, low-pass, high-pass, and echo, but WavMark is removed by Gaussian noise and MP3 without quality loss.
- Black-box watermark forgery and no-box forgery are largely ineffective (FPR near 0), while white-box forgery succeeds.
Limitations
- AudioMarkData covers 25 languages and only 4 age groups, limited by the Common Voice subsample.
- Black-box and white-box experiments use only 200 samples per method due to compute limits, so language-level analysis is underpowered (4 samples per language).
Experimental setup
Datasets: Common Voice LibriSpeech FMA
Benchmarks: AudioMarkData LibriSpeech FMA
Metrics: FNR FPR SNR ViSQOL
Baselines: AudioSeal AudioSeal-B Timbre WavMark
Builds on: HopSkipJumpAttack Square attack I-FGSM AudioSeal WavMark
Closest papers by meaning
- XAttnMark: Learning Robust Audio Watermarking with Cross-Attention (ICML 2025 · similarity 0.90)
- Speech Watermarking with Discrete Intermediate Representations (AAAI 2025 · similarity 0.89)
- Proactive Detection of Voice Cloning with Localized Watermarking (ICML 2024 · similarity 0.88)
- Robust Distortion-Free Watermark for Autoregressive Audio Generation Models (NeurIPS 2025 · similarity 0.88)
- Unbiased Watermark for Large Language Models (ICLR 2024 · similarity 0.86)
- Asymmetric Phase Coding Audio Watermarking (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.