Research map / Large language models / Language, safety and interpretability
An End-to-End Model for Logits-Based Large Language Models Watermarking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an end-to-end trained logits-perturbation watermark (encoder, decoder, and online text editor jointly optimized) that uses the on-the-fly LLM as a differentiable surrogate for editing and semantic loss, reporting better robustness-quality trade-offs than distortion-free methods.
Problem
LLM watermarks lose detectability when text is edited and logit or sampling biases degrade downstream quality; existing learned watermarks train encoder and decoder separately, missing joint optimization of robustness and text quality.
Approach
Jointly train a lightweight watermark encoder (MLP over top-k candidate contexts) and LSTM decoder end-to-end, using Gumbel-Softmax for differentiable sampling and the on-the-fly LLM itself for paraphrase-style text editing and semantic embedding loss via prompting, avoiding external models.
Key results
- Outperforms DiPmark by 37-39% F1 under paraphrasing and 17.2% average robustness, with PPL on par with distortion-free methods.
- Trained only on OPT-1.3B, achieves average F1 0.969 across attacks on Llama2-7B and generalizes zero-shot to Mixtral, Llama3 and Llama3.2 with average F1 0.969 on extra LLMs.
- Machine translation BLEU 31.062 versus 28.949 for the next best method (Unbiased) on NLLB-600M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use end-to-end watermarking rather than DiPmark | Llama2-7B watermarking · Llama2-7B paragraphing | F1: 0.670 -> 0.916 | Helped |
| use end-to-end watermarking rather than DiPmark | Llama2-7B watermarking · Llama2-7B copy-paste attack | F1: 0.915 -> 0.978 | Helped |
| use end-to-end watermarking rather than DiPmark | Qwen2.5-7B watermarking · Qwen2.5-7B paragraphing | F1: 0.942 -> 0.945 | No clear effect |
| use end-to-end watermarking rather than the strongest baseline | WMT16 German-English translation · WMT16 German-English | BLEU: 28.949 -> 31.062 | Helped |
| use watermarking rather than no watermark | Starcoder code generation · HumanEval | pass@1: 43.0 -> 34.0 | Hurt |
| add online editor N during training (ablation) | Ours watermarking · Llama2-7B paragraphing | F1: 0.867 -> 0.916 | Helped |
| add online editor N during training (ablation) | Ours watermarking · Llama2-7B copy-paste attack | F1: 0.952 -> 0.978 | Helped |
| add semantic loss L_sem (ablation) | Ours watermarking · Llama2-7B watermarked text | PPL: 9.561 -> 7.730 | Helped |
| lower watermark strength δ from 1.25 to .75 (ablation) | Ours watermarking · Llama2-7B paragraphing | F1: 0.916 -> 0.854 | Hurt |
| lower watermark strength δ from 1.25 to .75 (ablation) | Ours watermarking · Llama2-7B watermarked text | PPL: 7.730 -> 7.057 | Helped |
| increase watermark strength δ from 1.25 to 1.5 (ablation) | Ours watermarking · Llama2-7B paragraphing | F1: 0.916 -> 0.962 | Helped |
| increase watermark strength δ from 1.25 to 1.5 (ablation) | Ours watermarking · Llama2-7B watermarked text | PPL: 7.730 -> 8.566 | Hurt |
Surprising findings
- Lower temperature increases the advantage of the method's top-k logit perturbation, since the LLM concentrates on high-logit tokens.
- Increasing the watermark strength delta trades off PPL for robustness, and the method's CI-scored watermark is highly vulnerable to spoofing (TPR 0.55) until fine-tuned with a key-driven bias.
Limitations
- Training takes about 5 days on one GPU, longer than other learned watermarks such as SIR, UPV, and TSW.
- Watermark is vulnerable to spoofing/theft under the CI scoring (TPR 0.55), requiring fine-tuning with key-driven bias to mitigate.
- Cross-LLM converter relies on token alignment (about 73.4% average), so performance depends on tokenizer compatibility.
Experimental setup
Datasets: WikiText-103 C4
Benchmarks: C4 WMT16 German-English HumanEval
Metrics: F1 TPR at 1% FPR PPL BLEU pass@1
Baselines: KGW Unigram Unbiased DiPmark SIR SWEET TSW
Builds on: KGW Unigram DiPmark SIR Gumbel-Softmax
Closest papers by meaning
- PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMs (NeurIPS 2026 · similarity 0.93)
- Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach (NeurIPS 2025 · similarity 0.92)
- Provable Robust Watermarking for AI-Generated Text (ICLR 2024 · similarity 0.91)
- Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.90)
- SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders (NeurIPS 2025 · similarity 0.90)
- WaterMax: breaking the LLM watermark detectability-robustness-quality trade-off (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.