Research map / Large language models / Language, safety and interpretability
Optimizing Watermarks for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates the identifiability-vs-text-quality trade-off of green-red LLM watermarks as a multi-objective optimization and derives Pareto-optimal "OPT" watermarks that outperform KGW, DiP and ITS on the test-text trade-off.
Problem
Watermarks for LLM-generated text trade off identifiability (detection power) against degradation of text quality; there is no systematic way to optimize this trade-off within the class of robust, efficient green-red split watermarks.
Approach
Translate the test-text trade-off into a bi-objective problem (maximize expected green-list count, minimize expected log-perplexity shift) over a general class of shift functions; the Pareto-optimal solution shifts probability fully to green tokens whenever the per-token expected damage B is below a threshold beta, and leaves the distribution unchanged otherwise.
Key results
- OPT watermark attains the derived Pareto optimal bound and outperforms KGW, DiP and ITS on the expected green-count vs log-perplexity trade-off with OPT-1.3B.
- On summarization (BART-large) and translation (mBART), KGW outperforms OPT on ROUGE-1 vs power of the test, showing optimality for one text-quality metric does not transfer to another.
- The binomial assumption behind the power-of-test optimality breaks down: green-token events are positively correlated, with KL divergence increasing with watermark strength.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OPT instead of KGW, DiP, or ITS | OPT-1.3B watermarking · C4 | test-text trade-off using expected green-list tokens and expected log-perplexity | Helped |
| Use OPT instead of KGW | BART-large watermarking · CNN-DM text summarization | test-text trade-off using test power and ROUGE-1 | Hurt |
| Use OPT instead of KGW | Multilingual BART watermarking · WMT 2016 English-to-Romanian | test-text trade-off using test power and ROUGE-1 | Hurt |
| Use OPT instead of OPT′ | OPT watermarking · OPT-350m | test-text trade-off using expected log-perplexity | No clear effect |
| Use OPT′ instead of OPT | OPT watermarking · OPT-350m | test-text trade-off using expected log-perplexity squared plus between-token var | Helped |
| Apply KGW or OPT watermarking | LLM text generation · C4 | bigram and trigram diversity | Hurt |
| Use OPT rather than other watermarks | LLM text generation · C4 | bigram and trigram diversity | Hurt |
| Use DiP rather than the original LLM | LLM text generation · C4 | bigram and trigram diversity | Hurt |
| Use DiP rather than the original LLM | LLM text generation · CNN-DM summarization and WMT 2016 translation | identifiability | Hurt |
| Use ITS with beam search | LLM text generation · CNN-DM summarization and WMT 2016 translation | ROUGE-1 | Hurt |
| Use OPT′ instead of OPT | LLM text generation · OPT-350m | expected log-perplexity | No clear effect |
| Use stronger watermarks | KGW and OPT · C4 | dependence between green-token events | Hurt |
Surprising findings
- OPT watermark is outperformed by KGW on ROUGE-1-based text quality for summarization and translation, so optimized watermarks may not be superior under other metrics.
- OPT reduces bi- and tri-gram diversity more than other watermarks, a potential disadvantage.
Limitations
- Binomial independence assumption is violated for stronger watermarks, undermining the Pareto optimality guarantee for power of the test.
- Expected green-count and log-perplexity are used as proxies; OPT underperforms KGW on ROUGE-1 for summarization and translation.
- Experiments are limited to short sequences (T=30) and small OPT models.
Experimental setup
Datasets: C4 CNN-DM WMT 2016
Benchmarks: C4 news continuation CNN-DM WMT 2016 English-Romanian
Metrics: expected green-list tokens log-perplexity power of the test ROUGE-1 n-gram diversity KL divergence
Baselines: KGW hard watermark DiP ITS
Builds on: KGW watermark distribution-preserving watermark inverse transform sampling watermark
Closest papers by meaning
- Provable Robust Watermarking for AI-Generated Text (ICLR 2024 · similarity 0.90)
- Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach (NeurIPS 2025 · similarity 0.90)
- Unbiased Watermark for Large Language Models (ICLR 2024 · similarity 0.89)
- Learning to Watermark: A Selective Watermarking Framework for Large Language Models via Multi-Objective Optimization (NeurIPS 2025 · similarity 0.88)
- Token-Specific Watermarking with Enhanced Detectability and Semantic Coherence for Large Language Models (ICML 2024 · similarity 0.88)
- Inevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.