Research map / Large language models / Language, safety and interpretability
A Resilient and Accessible Distribution-Preserving Watermark for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiPmark, a permutation-based DiP-reweight watermark for LLMs that provably preserves the token distribution, is detectable without the LM or prompt, and is provably robust to text modifications with a guaranteed FPR.
Problem
Existing LLM watermarks either distort the sampling distribution, lack robustness to edits, need thousands of resampling steps or LM API/prompt access at detection, and use z-test statistics with imprecise false-positive guarantees.
Approach
Reweight token probabilities over a random permutation (cipher) by mixing two probability-interval reweights (alpha and 1-alpha) so the expected watermarked distribution equals the original; detection uses an exact binomial-based green token ratio with a KL-based concentration bound and a certified radius under arbitrary edits.
Key results
- DiPmark (alpha=0.45) keeps BLEU and summarization perplexity at the no-watermark level (e.g., BLEU 21.9 vs 21.8; perplexity 5.024 vs 5.021) while Soft watermark with delta=2.0 drops BLEU to 19.4 and raises perplexity to 6.241.
- Detecting 1,000 LLaMA-2 watermarked sequences takes 90 s with no LM or prompt access, at least 4x faster than ITS (412 s, requires LM access) and far faster than delta-watermark (12 h).
- Under random text modification at epsilon=0.3, DiPmark AUC is 0.8060 vs Soft (delta=1.5) 0.8033; on GPT-4 (gpt-4-0613, 200 tokens) 97 of 100 sequences are detected at FPR below 1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DiPmark (α=0.45) instead of no watermark | MBart · WMT'14 En-Ro test | BLEU: 21.8±0.3 -> 21.9±0.3 | No clear effect |
| Apply Soft watermark (δ=2.0) instead of no watermark | MBart · WMT'14 En-Ro test | BLEU: 21.8±0.3 -> 19.4±0.3 | Hurt |
| Apply DiPmark (α=0.45) instead of no watermark | BART-large · CNN-DM test | Perplexity: 5.021±0.018 -> 5.024±0.018 | No clear effect |
| Apply Soft watermark (δ=2.0) instead of no watermark | BART-large · CNN-DM test | Perplexity: 5.021±0.018 -> 6.241±0.023 | Hurt |
| Use DiPmark (α=0.45) instead of Soft watermark | LLaMA-2 text generation · 1,000 sequences | detection time: 92s -> 90s | No clear effect |
| Use DiPmark (α=0.45) rather than Soft watermark | LLaMA-2 text generation · random modification ε=0.2 | AUC: 0.9521 -> 0.9515 | No clear effect |
| Use DiPmark (α=0.45) rather than Soft watermark | LLaMA-2 text generation · paraphrasing ε=0.3 | AUC: 0.8157 -> 0.8216 | Helped |
| Increase α from 0.45 to 0.5 (ablation) | DiPmark on LLaMA-2 · text generation, z=1.517/√n | TPR: 0.9794 -> 0.9827 | No clear effect |
| Increase α from 0.45 to 0.5 (ablation) | DiPmark on LLaMA-2 · text generation, z=1.073/√n | TPR: 1.0000 -> 1.0000 | No clear effect |
| Increase α from 0.3 to 0.35 (ablation) | DiPmark on MBart · WMT'14 En-Ro test | BLEU: 22.0±0.3 -> 22.1±0.3 | No clear effect |
| Increase α from 0.45 to 0.5 (ablation) | DiPmark on MBart · WMT'14 En-Ro test | BLEU: 21.9±0.3 -> 21.8±0.3 | No clear effect |
| Increase α from 0.3 to 0.35 (ablation) | DiPmark on BART-large · CNN-DM test | Perplexity: 5.014±0.018 -> 4.998±0.018 | No clear effect |
Surprising findings
- The standard z-test used by the red-green watermark has empirical FPR (11.2% at nominal 10%) above its nominal level, so the authors switch to an exact binomial-based detector.
Limitations
- Detectability is slightly lower than the Soft watermark at equal text quality in some settings, and the authors note future work is needed to enhance detectability.
- GPT-4 case study treats unseen tokens as zero probability because only top-5 log-probs are exposed, so it is an approximation.
- Certified robustness is only for a fixed FPR and scales with the sequence length and bounded edit budget; results are shown for edits up to 30%.
Experimental setup
Datasets: WMT'14 En-Ro CNN-DM
Benchmarks: WMT'14 En-Ro CNN-DM
Metrics: BERTScore BLEU ROUGE perplexity AUC FPR FNR TPR TNR detection time
Baselines: Soft watermark (delta=1.0, 1.5, 2.0) ITS delta-watermark z-test detector
Builds on: Kirchenbauer et al. red-green list watermark (Soft watermark) Kuditipudi et al. inverse-sampling watermark (ITS) delta-reweight unbiased watermark SHA-256 hashing
Closest papers by meaning
- Inevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Models (NeurIPS 2024 · similarity 0.91)
- Break the Trade-off Between Watermark Strength and Speculative Sampling Efficiency for Language Models (ICLR 2026 · similarity 0.90)
- Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach (NeurIPS 2025 · similarity 0.90)
- Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.89)
- BiMark: Unbiased Multilayer Watermarking for Large Language Models (ICML 2025 · similarity 0.89)
- GaussMark: A Practical Approach for Structural Watermarking of Language Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.