Research map / Large language models / Language, safety and interpretability
Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a unified theoretical framework that jointly optimizes LLM watermark scheme and detector under worst-case FPR and distortion constraints, and a distortion-free, distribution-adaptive algorithm (DAWA) using a surrogate model and Gumbel-Max sampling.
Problem
Existing LLM watermarks lack formal optimality guarantees and design the scheme and detector separately, struggling to keep TPR high at ultra-low false positive rates while remaining distortion-free and model-agnostic.
Approach
Derives the universally minimum Type-II error under worst-case Type-I and distortion constraints, showing the optimal watermark sampling distribution should adapt to the LLM's next-token distribution; implements this at token level with a surrogate LM at detection and Gumbel-Max sampling of the auxiliary sequence.
Key results
- On Llama2-13B with C4 clean text, DAWA reaches ROC-AUC 0.999 and TPR@1%FPR 0.998, vs. 0.995/0.991 for KGW+23.
- On ELI5 (low-entropy QA) under token replacement attack with Mistral-8x7B, DAWA achieves ROC-AUC 0.993 and TPR@1%FPR 0.991, versus 0.983/0.932 for EXP-edit.
- DAWA's watermarked text BLEU is 0.214 and perplexity 10.034, close to the unwatermarked text (perplexity 10.020) and human text BLEU 0.219.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DAWA instead of the strongest listed baseline | Llama2-13B watermark detection · C4 clean | ROC-AUC: 0.996 -> 0.999 | Helped |
| Use DAWA instead of the strongest listed baseline | Llama2-13B watermark detection · ELI5 clean | ROC-AUC: 0.999 -> 0.998 | Hurt |
| Use DAWA instead of the strongest listed baseline | Llama2-13B watermark detection under token replacement · C4 | ROC-AUC: 0.973 -> 0.989 | Helped |
| Use DAWA instead of the strongest listed baseline | Llama2-13B watermark detection under token replacement · ELI5 | ROC-AUC: 0.973 -> 0.995 | Helped |
| Use DAWA instead of the strongest listed baseline | Mistral-8×7B watermark detection · C4 clean | ROC-AUC: 0.998 -> 0.999 | Helped |
| Use DAWA instead of the strongest listed baseline | Mistral-8×7B watermark detection · ELI5 clean | ROC-AUC: 0.999 -> 0.999 | No clear effect |
| Use DAWA instead of the strongest listed baseline | Mistral-8×7B watermark detection under token replacement · C4 | ROC-AUC: 0.980 -> 0.990 | Helped |
| Use DAWA instead of the strongest listed baseline | Mistral-8×7B watermark detection under token replacement · ELI5 | ROC-AUC: 0.983 -> 0.993 | Helped |
| Use DAWA instead of the strongest listed baseline | Llama2-13B watermark detection at 1% FP · C4 clean | TP@1% FP: 0.993 -> 0.998 | Helped |
| Use DAWA instead of the strongest listed baseline | Llama2-13B watermark detection at 1% FP · ELI5 clean | TP@1% FP: 0.991 -> 0.997 | Helped |
| Use DAWA instead of the strongest listed baseline | Llama2-13B watermark detection at 1% FP under token replacement · C4 | TP@1% FP: 0.858 -> 0.860 | Helped |
| Use DAWA instead of the strongest listed baseline | Llama2-13B watermark detection at 1% FP under token replacement · ELI5 | TP@1% FP: 0.892 -> 0.969 | Helped |
Surprising findings
- HCW+23 degrades sharply under token replacement attacks (e.g., TPR@1%FPR 0.268 on Llama2-13B C4), while other methods stay higher.
Limitations
- Theoretical optimality holds for fixed-length sequences; token-level scheme is only approximately optimal at sequence level.
- Detection relies on a surrogate model's approximation of the NTP distribution, and the auxiliary sequence is only incompletely recovered in practice.
- Evaluation limited to two LLMs and a few datasets, with token replacement attack as main robustness test.
Experimental setup
Datasets: C4 realnewslike ELI5 Wikipedia WMT19
Benchmarks: C4 realnewslike ELI5 Wikipedia WMT19
Metrics: ROC-AUC TPR at 1% FPR TPR at 10% FPR TPR at low FPR (1e-5 to 1e-1) perplexity BLEU
Baselines: KGW+23 EXP-edit Gumbel-Max HCW+23
Builds on: Gumbel-Max watermark green-red list watermark (KGW) inverse transform watermark
Closest papers by meaning
- PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMs (NeurIPS 2026 · similarity 0.92)
- An End-to-End Model for Logits-Based Large Language Models Watermarking (ICML 2025 · similarity 0.92)
- Learning to Watermark: A Selective Watermarking Framework for Large Language Models via Multi-Objective Optimization (NeurIPS 2025 · similarity 0.92)
- Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.92)
- Adaptive Text Watermark for Large Language Models (ICML 2024 · similarity 0.91)
- Provable Robust Watermarking for AI-Generated Text (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.