Research map / Image generation / Autoregressive and latent generation
Robust Distortion-Free Watermark for Autoregressive Audio Generation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Aligned-IS, a clustering-based distortion-free statistical watermark for autoregressive speech models that counters retokenization mismatch by treating tokens in the same k-means cluster as equivalent.
Problem
Statistical watermarks for autoregressive audio generation lose detectability because re-encoding generated audio into tokens (retokenization mismatch) does not reproduce the original token sequence.
Approach
Cluster audio tokens with k-means (h=20 clusters), then apply inverse-sampling reweighting aligned so each cluster occupies an equal-width interval in [0,1]; detection scores whether the observed token's cluster matches the pseudo-random number, so mismatched tokens within the same cluster still carry the signal. The method is provably distortion-free and gives a Hoeffding-based FPR bound.
Key results
- On SpiritLM (Longform QA-type settings), Aligned-IS reaches TPR@1% FPR of about 0.92-0.97 on MMW and Dolly CW, at least ~10% above other distortion-free baselines such as gamma-reweight, DiPmark and ITS.
- On SpeechGPT Longform QA, Aligned-IS achieves TPR@1% FPR of 0.95 versus 0.46 for gamma-reweight and 0.46 for DiPmark(0.4).
- Under 13 attacks, Aligned-IS keeps TPR@1% FPR above 0.8 on most codec and quantization attacks (e.g., 0.92 on EnCodec, 0.88 on 64-level quantization) while AudioSeal and WavMark fail on several of them.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use clustered aligned inverse sampling rather than the strongest biased baseline KGW (δ=2.0) | Audio watermark detection with SpiritLM · MMW Fake News | TPR@1% FPR: 0.92 -> 0.97 | Helped |
| Use clustered aligned inverse sampling rather than KGW (δ=2.0) | Audio watermark detection with SpeechGPT · Dolly CW | TPR@1% FPR: 0.73 -> 0.82 | Helped |
| Use clustered aligned inverse sampling rather than KGW (δ=2.0) | Audio watermark detection with SpeechGPT · Longform QA | TPR@1% FPR: 0.84 -> 0.95 | Helped |
| Use clustering to group tokens before watermarking (ablation) | SpiritLM audio token sequences · MMW Book Report | Mismatch rate: 0.3749 -> 0.2117 | Helped |
| Use 20 clusters rather than 10 clusters (ablation) | Aligned-IS on SpiritLM · Dolly CW | TPR@FPR=1%: 0.798343 -> 0.922018 | Helped |
| Use 30 clusters rather than 20 clusters (ablation) | Aligned-IS on SpiritLM · Dolly CW | TPR@FPR=1%: 0.922018 -> 0.724234 | Hurt |
| Use longer audio segments for detection (ablation) | Aligned-IS on SpeechGPT · Longform QA | TPR@FPR=1%: 0.5000 -> 0.9451 | Helped |
| Apply a 320-sample time shift (ablation) | Aligned-IS on SpiritLM · Longform QA | TPR@FPR=1%: 0.94 -> 0.43 | Hurt |
| Add Aligned-IS watermarking rather than leave audio unwatermarked | SpiritLM audio generation · Dolly CW | FAD (lower is better): 0.0487 -> 0.0512 | No clear effect |
| Add Aligned-IS watermarking rather than leave audio unwatermarked | SpiritLM audio generation · LibriSpeech | Speaker Similarity: 0.7550 -> 0.7548 | No clear effect |
Surprising findings
- Post hoc watermarks AudioSeal and WavMark are highly fragile: AudioSeal drops to 0.00 under EnCodec and quantization and to 0.00 under speed changes, while WavMark fails on low-bitrate Opus.
- Aligned-IS is weak under speed perturbation (TPR about 0.2-0.4) because token-frame alignment is broken by time-scaling.
Limitations
- No formal guarantee that clustering captures all retokenization errors, especially for new audio patterns.
- Requires per-model clustering of audio token embeddings before use.
- Not robust to speed or time-misalignment attacks; when clusters have probability below 1/h, detection accuracy can drop.
Experimental setup
Datasets: MMW Book Report MMW Story MMW Fake News Dolly CW Longform QA Finance QA WaterBench LibriSpeech
Benchmarks: MMW Book Report MMW Story MMW Fake News Dolly CW Longform QA Finance QA LibriSpeech
Metrics: TPR@FPR=1% TPR@FPR=0.1% median p-value FAD NISQA DNSMOSPro speaker similarity ASR-CER ASR-WER
Baselines: KGW Unigram watermark gamma-reweight DiPmark ITS-edit AudioSeal WavMark
Builds on: KGW inverse sampling (Kuditipudi et al.) DiPmark ITS SynthID
Closest papers by meaning
- AudioMarkBench: Benchmarking Robustness of Audio Watermarking (NeurIPS 2024 · similarity 0.88)
- Watermarking Autoregressive Image Generation (NeurIPS 2025 · similarity 0.87)
- XAttnMark: Learning Robust Audio Watermarking with Cross-Attention (ICML 2025 · similarity 0.86)
- Asymmetric Phase Coding Audio Watermarking (NeurIPS 2026 · similarity 0.86)
- Speech Watermarking with Discrete Intermediate Representations (AAAI 2025 · similarity 0.85)
- Masked Audio Generation using a Single Non-Autoregressive Transformer (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.