Research map / Large language models / Language, safety and interpretability
De-mark: Watermark Removal in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes De-mark, a query-based framework that steals n-gram watermark parameters (prefix length, strength, red/green lists) via random selection probing and removes or exploits them, with a total-variation bound on the post-removal distribution.
Problem
N-gram watermarks for LLM-generated text are assumed robust, but their parameters can be stolen and removed; prior stealing methods need the hash function and paraphrasing, which distorts the LM distribution.
Approach
Random selection probing: prompt the LM to randomly pick between two phrases; the log-ratio of selection probabilities is an unbiased estimator of the watermark strength delta for token pairs in different red/green lists. This reveals h, delta, and the green list, and a logit reweighting then removes the watermark while bounding the TV distance from the original distribution.
Key results
- On Llama3.1 8B with MMW Book Report, De-mark reduces TPR@FPR=0.1% from 89% to 2% while GPT score stays near 94.8 to 94.6.
- Red/green list identification reaches roughly 85-87% accuracy in both gray-box and black-box settings.
- Watermark exploitation with a Llama3.2-1B attacker achieves 93% TPR@0.1% FPR in gray-box mode with GPT score similar to unwatermarked text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply De-mark to remove the watermark | Llama3.1 8B + Watermark · MMW Book Report | TPR@FPR=0.1%: 89% -> 2% | Helped |
| Apply De-mark to remove the watermark | Llama3.1 8B + Watermark · Dolly CW | TPR@FPR=0.1%: 92% -> 10% | Helped |
| Apply De-mark to remove the watermark | Mistral 7B + Watermark · MMW Book Report | TPR@FPR=0.1%: 95% -> 15% | Helped |
| Apply De-mark to remove the watermark | Mistral 7B + Watermark · Dolly CW | TPR@FPR=0.1%: 87% -> 13% | Helped |
| Apply De-mark to remove the watermark | Llama3.2 3B + Watermark · MMW Book Report | TPR@FPR=0.1%: 91% -> 6% | Helped |
| Apply De-mark to remove the watermark | Llama3.2 3B + Watermark · Dolly CW | TPR@FPR=0.1%: 82% -> 9% | Helped |
| Apply De-mark to remove the watermark | ChatGPT watermarked with n-gram watermark · ChatGPT | TPR@FPR=0.1%: 89.58% -> 25.00% | Helped |
| Increase removal strength η from 1.0 to 1.5 (ablation) | De-mark on Llama3.1 8B · MMW Story | TPR@FPR=0.1%: 6.25% -> 0.00% | Helped |
| Increase removal strength η from 1.5 to 2.0 (ablation) | De-mark on Llama3.1 8B · MMW Story | GPT Score: 90.40 -> 90.02 | Hurt |
| Increase removal strength η from 1.0 to 1.5 (ablation) | De-mark on Mistral 7B · MMW Story | TPR@FPR=0.1%: 26.04% -> 4.17% | Helped |
| Increase removal strength η from 1.5 to 2.0 (ablation) | De-mark on Mistral 7B · MMW Story | TPR@FPR=0.1%: 4.17% -> 1.04% | Helped |
| Increase removal strength η from 1.0 to 1.5 (ablation) | De-mark on Llama3.2 3B · MMW Story | TPR@FPR=0.1%: 17.71% -> 0.00% | Helped |
Surprising findings
- Black-box estimation with only 10 samples per probability still recovers red/green lists at over 85% accuracy.
Limitations
- Requires access to top-k token log-probabilities (gray-box); black-box variant needs many repeated queries.
- Estimation of delta is imprecise when watermark strength is small.
- Removal strength eta trades off against generation quality (higher eta lowers GPT score).
Experimental setup
Datasets: Dolly creative writing MMW Book Report MMW Story WaterBench
Benchmarks: MMW Book Report Dolly CW MMW Story WaterBench (Entity Probing, Concept Probing, Long-form QA)
Metrics: TPR@FPR median p-value GPT score F1 precision recall accuracy Rouge-L
Baselines: Watermark stealing (Jovanović et al.) gamma-reweight DiPmark
Builds on: n-gram watermarking (Kirchenbauer-style red-green lists) Jovanović et al. watermark stealing
Closest papers by meaning
- On the Learnability of Watermarks for Language Models (ICLR 2024 · similarity 0.88)
- Majority Bit-Aware Watermarking for Large Language Models (NeurIPS 2026 · similarity 0.88)
- Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.88)
- Learning to Watermark: A Selective Watermarking Framework for Large Language Models via Multi-Objective Optimization (NeurIPS 2025 · similarity 0.87)
- StealthInk: A Multi-bit and Stealthy Watermark for Large Language Models (ICML 2025 · similarity 0.87)
- Edit Distance Robust Watermarks via Indexing Pseudorandom Codes (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.