Research map / Large language models / Language, safety and interpretability
Watermark Stealing in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that querying a watermarked LLM's API lets an attacker automatically steal approximate watermark rules, enabling cheap spoofing and greatly boosted scrubbing of KGW-family schemes, including KGW2-SelfHash.
Problem
LLM text watermarks were considered fit for deployment and robust to adversaries, but their vulnerability to attackers who reverse-engineer the secret watermarking rules via API queries (spoofing and scrubbing) was poorly understood, especially for the state-of-the-art KGW2-SelfHash scheme.
Approach
Introduces watermark stealing: an automated algorithm that estimates per-context green-token scores from watermarked API responses versus an auxiliary model's base distribution, then uses these scores to steer generation for spoofing (promote green tokens) and scrubbing (demote them), with partial-context heuristics to handle sparsity.
Key results
- Spoofing KGW2-SelfHash with Mistral-7B as attacker achieves FPR*@1e-3 around 0.81 to 0.95 across tested settings, with query cost under $50 (30,000 queries).
- Boosting DIPPER with the stolen scores raises scrubbing success on KGW2-SelfHash from about 0-2% to 79-90% on average, versus at most ~30% for baselines.
- Removing partial-context scores reduces average spoofing success from around 80% to below 50% at n=30,000 queries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use stolen watermark scores to guide Dipper paraphrasing | KGW2-SelfHash scrubbing · Dolly CW | FNR* @1e-3: 0.02 -> 0.90 | Helped |
| Use stolen watermark scores to guide Dipper paraphrasing | KGW2-SelfHash scrubbing · Reddit WritingPrompts | FNR* @1e-3: 0.00 -> 0.79 | Helped |
| Use stolen watermark scores to guide Pegasus paraphrasing | KGW2-SelfHash scrubbing · Dolly CW | FNR* @1e-3: 0.15 -> 0.84 | Helped |
| Use stolen watermark scores to guide Dipper paraphrasing | KGW2-Sum scrubbing · Dolly CW | FNR* @1e-3: 0.79 -> 0.98 | Helped |
| Use stolen watermark scores to guide Dipper paraphrasing | KGW-Soft scrubbing · Dolly CW | FNR* @1e-3: 0.26 -> 0.85 | Helped |
| Remove partial-context scores from the watermark-stealing attacker (ablation) | KGW2-SelfHash spoofing · Dolly CW, Llama2-7B owner, n=30,000 queries | FPR* @1e-6: around 80% to below 50% | Hurt |
| Use Mistral-7B to spoof KGW2-SelfHash | KGW2-SelfHash · Dolly CW, Llama-7B owner | FPR* @1e-3: 0.81 | Helped |
| Use KGW2-Sum instead of KGW2-SelfHash | watermark spoofing · Dolly CW, Llama-7B owner | FPR* @1e-3: 0.81 -> 0.54 | Hurt |
Surprising findings
- Stealing the watermark also strongly boosts scrubbing, contrary to the belief that spoofing and scrubbing robustness trade off against each other.
- KGW2-SelfHash, previously considered safe, can be spoofed with high success in realistic chat-model settings at low FPR.
Limitations
- Results are reported for the most restrictive setting; attack success depends on the attacker's auxiliary model and base-distribution estimates.
- Scoring normalization and heuristic partial-context weights are hand-set and may be suboptimal.
- Evaluation focuses on distribution-modifying schemes of the KGW family and does not cover other watermark classes.
Experimental setup
Datasets: C4 RealNewsLike Dolly-CW HarmfulQ AdvBench MMW BookReports MMW FakeNews Reddit WritingPrompts
Benchmarks: Dolly-CW HarmfulQ+AdvBench MMW BookReports MMW FakeNews Reddit WritingPrompts
Metrics: FPR*@1e-3 (spoofing success) FNR*@1e-3 (scrubbing success) median p-value perplexity GPT-4 quality score P-SP score
Baselines: DIPPER Pegasus paraphraser ChatGPT paraphrasing Recursive DIPPER Unigram KGW-Soft KGW-Hard KGW2-Sum
Builds on: KGW watermark SelfHash DIPPER paraphraser
Closest papers by meaning
- Discovering Spoofing Attempts on Language Model Watermarks (ICML 2025 · similarity 0.93)
- Enhancing LLM Watermark Resilience Against Both Scrubbing and Spoofing Attacks (NeurIPS 2025 · similarity 0.92)
- No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choices (NeurIPS 2024 · similarity 0.91)
- Watermarks in the Sand: Impossibility of Strong Watermarking for Language Models (ICML 2024 · similarity 0.89)
- Can Watermarked LLMs be Identified by Users via Crafted Prompts? (ICLR 2025 · similarity 0.89)
- Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.