Research map / Probabilistic and scientific modeling / Molecules and proteins
Speech Watermarking with Discrete Intermediate Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiscreteWM, a frame-wise speech watermarking framework that embeds watermark bits by altering the parity of discrete VQ-VAE token IDs, using a manipulator model to choose imperceptible replacement tokens.
Problem
Speech watermarking methods embed messages in continuous latent space, trading off imperceptibility, robustness and capacity, and rely on fixed-length inputs with slow sliding-window localization for detection.
Approach
Embed watermark bits into the modular arithmetic (parity) of discrete VQ-VAE token IDs of speech, with a masked VQ-VAE for frame-level localization of reconstructed frames, a restorer for parity recovery, and a probability-based manipulator that selects replacement tokens; detection uses a one-proportion Z-test for utterance-level AI-generated content detection.
Key results
- Ours-32bps achieves mean BER 0.28% across distortions vs 2.58% for WavMark, with PESQ 4.45 vs 4.31.
- On AI-generated speech detection, DiscreteWM reaches RTF 0.0044 versus 0.1438 for WavMark (about 22.1x faster), with mean BER 0.32%.
- DiscreteWM encodes 1 to 150 bits per second of speech; at 150bps PESQ is 3.92 and mean BER 0.27%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use discrete rather than continuous intermediate representations (ablation) | DiscreteWM · Information hiding | Mean BER (%): 2.39 -> 0.28 | Helped |
| Add the probability-based manipulator instead of manually choosing random codes (ablation) | DiscreteWM · Information hiding | PESQ: 3.96 -> 4.45 | Helped |
| Increase encoding capacity from 10bps to 32bps (ablation) | DiscreteWM · Information hiding | PESQ: 4.47 -> 4.45 | Hurt |
| Increase encoding capacity from 32bps to 50bps (ablation) | DiscreteWM · Information hiding | PESQ: 4.45 -> 4.27 | Hurt |
| Increase encoding capacity from 32bps to 150bps (ablation) | DiscreteWM · Information hiding | PESQ: 4.45 -> 3.92 | Hurt |
| Repeat random watermark-position selection from 1 to 50 trials (ablation) | DiscreteWM · Information hiding | SNR: 37.93 -> 39.70 | Helped |
| Use waveform instead of spectrogram input (ablation) | DiscreteWM · AI-generated speech detection | Mean BER (%): 0.32 -> 0.53 | Hurt |
| Replace the Transformer manipulator with Conv1d (ablation) | DiscreteWM · Information hiding | Mean BER (%): 0.28 -> 0.54 | Hurt |
| Replace the Transformer manipulator with Conformer (ablation) | DiscreteWM · Information hiding | Mean BER (%): 0.28 -> 0.27 | Helped |
| Change codebook length from 128 to 24 (ablation) | DiscreteWM · Information hiding | PESQ: 4.45 -> 3.63 | Hurt |
| Change codebook length from 128 to 64 (ablation) | DiscreteWM · Information hiding | PESQ: 4.45 -> 4.18 | Hurt |
| Change codebook length from 128 to 256 (ablation) | DiscreteWM · Information hiding | Mean BER (%): 0.28 -> 0.36 | Hurt |
Surprising findings
- Operating on the magnitude spectrogram rather than the raw waveform gave slightly better imperceptibility and robustness than a waveform-based VQ variant (Encodec backbone).
Limitations
- Different codebook entries have different robustness and imperceptibility characteristics that the method does not analyze.
- Uses a GAN-based architecture; diffusion-based alternatives are not explored.
- Inference speed could be further improved with more efficient networks.
Experimental setup
Datasets: LibriTTS
Benchmarks: LibriTTS test-clean LibriTTS-based voice cloning test set (NaturalSpeech 2, Mega-TTS 2 outputs)
Metrics: BER PESQ SNR RTF Z-statistic
Baselines: Audiowmark DeAR Chang Liu's method WavMark SeamlessWM
Builds on: VQ-VAE Encodec WavMark HiFi-GAN
Closest papers by meaning
- AudioMarkBench: Benchmarking Robustness of Audio Watermarking (NeurIPS 2024 · similarity 0.89)
- Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.86)
- Proactive Detection of Voice Cloning with Localized Watermarking (ICML 2024 · similarity 0.85)
- ArcMark: Distortion-Free Multi-Byte LLM Watermark via Optimal Transport (NeurIPS 2026 · similarity 0.85)
- XAttnMark: Learning Robust Audio Watermarking with Cross-Attention (ICML 2025 · similarity 0.85)
- Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.