Research map / Large language models / Language, safety and interpretability
Token-Specific Watermarking with Enhanced Detectability and Semantic Coherence for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes token-specific watermarking for LLMs where two lightweight MLPs produce per-token splitting ratios and watermark logits, trained with multi-objective optimization (MGDA) over a differentiable z-score detection loss and a SimCSE semantic loss.
Problem
KGW-style inference-time watermarks use constant splitting ratio and watermark logit for all tokens, which trades detectability against semantic coherence of generated text.
Approach
Learn token-specific splitting ratios (gamma) and watermark logits (delta) from the preceding token's embedding via two small MLPs, using Gumbel-Softmax for differentiable green-list sampling and optimizing a relaxed z-score and SimCSE similarity jointly with MGDA to find Pareto-optimal trade-offs.
Key results
- On OPT-1.3B, the learned method improves the TPR vs. SimCSE Pareto frontier over KGW, SWEET, MultiBit, SIR and EXP-edit at 0% and 1% FPR.
- Against EXP-edit (Top-k=50), the method reaches TPR 1.000 at both 0% and 1% FPR with SimCSE 0.713 versus 0.677.
- Generation time on OPT-1.3B is 3.946 s versus 3.827 s for KGW and 24.693 s for EXP-edit, and the OPT-trained networks generalize to LLaMA-2 7B/13B/70B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use token-specific splitting ratios and watermark logits with multi-objective optimization instead of EXP-edit (Top-k=50) | OPT-1.3B watermarking · OPT-1.3B test, FPR 0% | TPR: 0.968 -> 1.000 | Helped |
| Use token-specific splitting ratios and watermark logits with multi-objective optimization instead of EXP-edit (Top-k=50) | OPT-1.3B watermarking · OPT-1.3B test | SimCSE: 0.677 -> 0.713 | Helped |
| Use token-specific watermarking with multi-objective optimization instead of EXP-edit | OPT-1.3B watermarking · OPT-1.3B test, FPR 1% | TPR: 0.996 -> 1.000 | Helped |
| Use token-specific watermarking with multi-objective optimization instead of EXP-edit | OPT-1.3B watermarking · OPT-1.3B test | SimCSE: 0.655 -> 0.713 | Helped |
| Use token-specific watermarking instead of EXP-edit (Top-k=50) | OPT-1.3B watermarking · OPT-1.3B, beam search (Num Beams=2) | PPL: 1.628 -> 1.847 | Hurt |
| Use token-specific watermarking instead of EXP-edit | OPT-1.3B generation · OPT-1.3B, 200-token generation | generation time (s): 24.693 -> 3.946 | Helped |
| Use token-specific watermarking instead of EXP-edit | OPT-1.3B detection · OPT-1.3B detection | detection time (s): 155.045 -> 0.166 | Helped |
| Use token-specific splitting ratios and watermark logits with multi-objective optimization instead of KGW | OPT-1.3B and LLAMA2 watermarking · OPT-1.3B and LLAMA2 (7B, 13B, 70B) | TPR and semantic similarity trade-off | Helped |
| Use token-specific watermarking instead of SWEET | LLM watermarking · FPR 0% | TPR and semantic similarity trade-off | Helped |
| Use token-specific watermarking instead of SWEET | LLM watermarking · FPR 1% | TPR and semantic similarity trade-off | No clear effect |
| Use token-specific watermarking instead of MultiBit | LLM watermarking · TPR and semantic similarity trade-off | TPR and semantic similarity trade-off | Helped |
| Use token-specific watermarking instead of SIR | LLM watermarking · TPR and semantic similarity trade-off | TPR and semantic similarity trade-off | Helped |
Surprising findings
- Learned splitting ratios and watermark logits are lower after adjectives and determiners and higher after punctuation, interpreted as tracking where the next token is constrained.
- SWEET fails to reach 100% TPR at 0% FPR even with high delta, because few tokens are high-entropy and watermarkable.
Limitations
- Requires the LLM embedding matrix at detection time, unlike the purely statistical KGW detector.
- Pareto frontiers are obtained by curve-fitting a handful of initializations, so the frontier is approximate.
- Detection is weaker when generated text is short, and the theoretical FPR approximation may be imprecise.
Experimental setup
Datasets: C4 news-like subset
Benchmarks: C4 news-like subset
Metrics: TPR at 0% FPR TPR at 1% FPR SimCSE similarity z-score perplexity (OPT-2.7B oracle) generation and detection time
Baselines: KGW SWEET MultiBit SIR EXP-edit SWEET NoPrompt SIR NoPrompt
Builds on: KGW SimCSE MGDA Gumbel-Softmax
Closest papers by meaning
- Learning to Watermark: A Selective Watermarking Framework for Large Language Models via Multi-Objective Optimization (NeurIPS 2025 · similarity 0.92)
- Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach (NeurIPS 2025 · similarity 0.90)
- Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.89)
- PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMs (NeurIPS 2026 · similarity 0.89)
- Adaptive Text Watermark for Large Language Models (ICML 2024 · similarity 0.89)
- A Semantic Invariant Robust Watermark for Large Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.