Research map / Large language models / Language, safety and interpretability
WaterMax: breaking the LLM watermark detectability-robustness-quality trade-off
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WaterMax, an LLM text watermark that leaves the model and sampling untouched, generates chunk drafts and keeps the one with the lowest p-value, achieving near-1 detectability at FPR 1e-6 with negligible perplexity loss.
Problem
Existing LLM watermarks trade detectability against text quality, since detectability depends on text entropy and strong watermarks degrade quality; robustness is also hard to characterize theoretically.
Approach
Design the detector first (sum of Gaussian token scores over chunks) and build the generator to maximize detection: at each chunk, sample n drafts, keep the one with the highest incremental score (greedy with m=1), so the LLM distribution and sampling are preserved while detectability grows with the number of drafts.
Key results
- WaterMax reaches detectability close to 1 at P_FA=1e-6 on Llama3-8b-Instruct while keeping relative perplexity nearly unchanged (b=6 shows virtually no loss).
- KGW at delta=3.0 has a significantly larger perplexity increase yet remains far less detectable than WaterMax.
- Under MMW attacks at low temperatures WaterMax is by far the most robust; the robust detector resists more synonym, misspelling and swap attacks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WaterMax chunk-based draft selection instead of token-level watermarking | Llama-3-8b-Instruct · Mark My Words benchmark across temperatures θ=0.8–1.0 | detectability | Helped |
| Use WaterMax with b=6 beam-search tokens | WaterMax · Mark My Words benchmark | relative perplexity | No clear effect |
| Increase beam-search tokens from b=0 to b=6 (ablation) | WaterMax (n=8, N=16) · Mark My Words benchmark, θ=1.0 | detectability | Helped |
| Increase beam-search tokens from b=0 to b=6 (ablation) | WaterMax (n=8, N=16) · Mark My Words benchmark, θ=1.0 | text quality | No clear effect |
| Use fewer than 6 beam-search tokens (1≤b<6) (ablation) | WaterMax (n=8, N=16) · Mark My Words benchmark, θ=1.0 | detectability | Helped |
| Use fewer than 6 beam-search tokens (1≤b<6) (ablation) | WaterMax (n=8, N=16) · Mark My Words benchmark, θ=1.0 | text quality | Hurt |
| Increase hashing window size h (ablation) | WaterMax · No attack | detectability | Helped |
| Increase hashing window size h (ablation) | WaterMax · Text modification attacks | robustness | Hurt |
| Set hashing window size to h=2 (ablation) | WaterMax · No attack, P_FA=10^-6 | detectability | Hurt |
| Set hashing window size to h≥4 (ablation) | WaterMax · No attack, P_FA=10^-6 | detectability: P_D=1.0 | Helped |
| Set hashing window size to h=6 (ablation) | WaterMax · No attack | match to theoretical score distribution | Helped |
| Use the robust global-score detector instead of the optimal chunk detector (ablation) | WaterMax · No attack | detectability | Hurt |
Surprising findings
- Aaronson's scheme is more robust than WaterMax for some attacks at temperatures starting from 1.0, attributed to high variance of its p-values.
- Pure WaterMax with no beam search (b=0) is not the best; enforcing diversity via beam search is needed for draft independence.
Limitations
- High computational cost: Nn text generations, with ~30 min vs ~6 min for baselines on 296 texts on one A100.
- Not distortion-free for n>2; the empirical quality claim relies on small chunk probabilities.
- Robustness analysis assumes Gaussian token scores and a model approximation for large N.
Experimental setup
Datasets: Mark My Words benchmark Wikipedia 2018 entries
Benchmarks: Mark My Words
Metrics: true-positive rate at FPR 1e-6 relative perplexity ROUGE-L detectability under attack
Baselines: KGW Aaronson's scheme binary inverse transform
Builds on: KGW Aaronson Gumbel watermark Mark My Words benchmark
Closest papers by meaning
- PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMs (NeurIPS 2026 · similarity 0.91)
- Provable Robust Watermarking for AI-Generated Text (ICLR 2024 · similarity 0.90)
- Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.90)
- Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach (NeurIPS 2025 · similarity 0.90)
- An End-to-End Model for Logits-Based Large Language Models Watermarking (ICML 2025 · similarity 0.90)
- Unbiased Watermark for Large Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.