Research map / Large language models / Language, safety and interpretability
An Unforgeable Publicly Verifiable Watermark for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UPV, a publicly verifiable LLM watermark where a neural watermark generator (sharing token embeddings) is paired with a separate LSTM detector that takes only text, so detection does not expose the generation key.
Problem
Existing LLM watermark detectors need the secret generation key, so public detection exposes the key and lets third parties forge watermarks.
Approach
Replace the shared secret key with two neural networks: a watermark generator that labels candidate tokens from a local window, and an LSTM detector trained on generator-labeled random token-ID sequences; the two share frozen token embeddings so the detector learns quickly, and the detector works on whole texts without per-token labels.
Key results
- Network-based (UPV) detection on GPT-2 with top-K sampling reaches 99.3 F1 on C4 versus 99.7 for key-based detection.
- Without shared embedding layers, detector F1 collapses (e.g., OPT 1.3B C4 F1 0.0 vs 97.5 with sharing), per the ablation.
- Reverse-training forgery attacks fall toward random guess (F1 about 0.5) as window size grows, and word-frequency cracking becomes ineffective at larger windows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the UPV network-based detector instead of key-based z-score detection | Watermarked text detection · C4 and DBpedia Class, Top-K sampling and Beam Search across GPT-2, OPT 1.3B, and LLaMA 7B | F1: GPT-2: 99.7 -> 99.3 (C4, Top-K); 99.9 -> 99.0 (DBpedia, Top-K); 99.8 -> 99.6 (C4, Beam); 99.9 -> 99.6 (DBpedia, Beam). O | Mixed |
| Add shared generator/detector embedding layers without fine-tuning (ablation) | UPV detector · C4 and DBpedia Class, Top-K sampling; GPT-2, OPT 1.3B, and LLaMA 7B | F1: GPT-2 C4: 0.5 -> 99.3; DBpedia: 47.5 -> 99.0. OPT 1.3B C4: 0.0 -> 97.5; DBpedia: 1.3 -> 98.0. LLaMA 7B C4: 31.4 -> 98.2; | Helped |
| Fine-tune the shared embedding layers during detector training (ablation) | UPV detector · C4 and DBpedia Class, Top-K sampling; GPT-2 and OPT 1.3B | F1: GPT-2 C4: 99.3 -> 90.9; DBpedia: 99.0 -> 92.5. OPT 1.3B C4: 97.5 -> 87.3; DBpedia: 98.0 -> 84.8 | Hurt |
| Fine-tune the shared embedding layers during detector training (ablation) | UPV detector · C4 and DBpedia Class, Top-K sampling; LLaMA 7B | F1: C4: 98.2 -> 87.8; DBpedia: 97.5 -> 79.6 | Hurt |
Surprising findings
- Fine-tuning the shared embedding layers during detector training reduced F1 (by about 11.1% on average), so keeping them frozen was best.
- Reverse-training from the detector to the generator gives near-random results at larger window sizes, which the authors take as evidence of computational asymmetry.
Limitations
- Robustness to multiple rewrites or stronger attacks is not evaluated; only a single GPT-3.5 rewrite is tested.
- Detector F1 trails the key-based upper bound, especially for OPT-1.3B top-K and LLaMA-7B top-K (e.g., 97.5 vs 99.8 F1).
Experimental setup
Datasets: C4 DBpedia Class WMT14
Benchmarks: C4 DBpedia Class WMT14
Metrics: F1 FPR FNR perplexity BLEU
Baselines: Key-based z-score watermark detection
Builds on: KGW green-red list watermark z-score test LSTM
Closest papers by meaning
- Provable Robust Watermarking for AI-Generated Text (ICLR 2024 · similarity 0.88)
- PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMs (NeurIPS 2026 · similarity 0.87)
- On the Learnability of Watermarks for Language Models (ICLR 2024 · similarity 0.87)
- Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.87)
- Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach (NeurIPS 2025 · similarity 0.87)
- Edit Distance Robust Watermarks via Indexing Pseudorandom Codes (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.