Research map / Architectures and efficiency / Pruning and efficient tuning
Task-Agnostic Language Model Watermarking via High Entropy Passthrough Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes task-agnostic blackbox LLM watermarking by inserting trainable "passthrough" layers that act as identity on clean inputs but output near-uniform (max-entropy) distributions when a private key is present, verified by entropy change.
Problem
Blackbox ownership verification for pre-trained language models is limited to classification tasks or requires poisoning during pretraining or fine-tuning datasets; a watermark is needed that works for both classification and sequence-to-sequence tasks, is robust to fine-tuning and pruning, and is c
Approach
Insert small passthrough layers into a frozen-ish pretrained network, trained with a self-supervised MSE identity loss on clean pretraining data plus a loss pushing outputs to uniform distribution when a private key is inserted; ownership is detected by the entropy difference between keyed and unkeyed prompts.
Key results
- PTL-135 on BERT averages 0.959 WACC and 0.021 FP across seven datasets with ACC 0.939 (vs. 0.939 unwatermarked-level), while Gu/M averages 0.357 WACC.
- On GPT-2, PTL-147 and PTL-13579 reach WACC 1.000 with FP 0.000 and LAMBADA accuracy 0.250/0.235 vs 0.225 baseline; the Chen et al. baseline reaches WACC 0.692.
- Passthrough watermarking on BERT trains in about 3.05-3.13 hours versus 14.03 hours for NeuBA, while keeping task accuracy comparable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add passthrough layers rather than watermarking all model parameters | BERT-based-uncased · Classification overall average | WACC: 0.915$\pm$0.025 -> 0.964$\pm$0.011 | Helped |
| add passthrough layers rather than watermarking all model parameters | BERT-based-uncased · Classification overall average | FP: 0.036$\pm$0.011 -> 0.027$\pm$0.007 | Helped |
| use PTL-147 rather than PTL-1 (ablation) | GPT-2 · LAMBADA | PPL: 96.9 -> 85.4 | Helped |
| use PTL-13579 rather than PTL-147 (ablation) | GPT-2 · BoolQ | EM: 0.480 -> 0.135 | Hurt |
| increase the number of passthrough layers from one to five (ablation) | GPT-2 · LAMBADA | ACC: 0.233 -> 0.235 | Mixed |
| increase the number of passthrough layers from one to five (ablation) | Llama2-7B · LAMBADA | ACC: 0.668 -> 0.654 | Hurt |
| increase the number of passthrough layers from one to five (ablation) | Llama2-7B · LAMBADA | FP: 0.015 -> 0.002 | Helped |
| add self-supervised passthrough loss terms (ablation) | GPT-2 watermarking · clean samples during watermark training | entropy convergence speed | Helped |
| remove all passthrough layers, then fine-tune | GPT-2 PTL-13579 · LAMBADA, layer removal + fine-tuning | ACC: 0.235 -> 0.073 | Hurt |
| remove all passthrough layers, then fine-tune | GPT-2 PTL-13579 · LAMBADA, layer removal + fine-tuning | WACC: 1.000 -> 0.954 | Hurt |
| increase passthrough layer count from PTL-1 to PTL-358 (ablation) | BERT-based-uncased · NG, fine-pruning | WACC: 0.494 -> 0.864 | Helped |
| increase passthrough layer count from PTL-1 to PTL-358 (ablation) | BERT-based-uncased · SST2, fine-pruning | WACC: 0.937 -> 1.000 | Helped |
Surprising findings
- Removing passthrough layers and fine-tuning (layer removal + finetuning attack) severely degrades the model, e.g. PTL-13579 loses about 16% LAMBADA accuracy and perplexity rises to 1562.
- Adding a single passthrough layer (PTL-1) gives high WACC on most tasks, suggesting additional layers mainly improve attack resilience rather than watermark accuracy.
Limitations
- Robustness to attacks is shown mainly for a limited set of attacks; the paper notes WACC drops with low fine-tuning epochs and for PTL-1 on NG under fine-pruning.
- Evaluated only on BERT, GPT-2 and Llama2-7B; applicability to industry-scale models is left to future work.
- Layer removal with finetuning can lower WACC by roughly 5-8% for some configurations, and PTL-1 shows FP 0.863 after the attack.
Experimental setup
Datasets: BookCorpus OpenWebText SST2 IMDB SNLI MNLI AGNews NewsGroup PAWS LAMBADA BoolQ SquadQA
Benchmarks: SST2 IMDB SNLI MNLI AGNews NewsGroup PAWS LAMBADA WikiText BoolQ SquadQA
Metrics: ACC WACC FP rate AUC runtime (hr) perplexity exact match contains
Baselines: Gu (Single Task) Gu (Multi-Task) NeuBA FullParam-BL Chen et al. (word2sentence)
Builds on: NeuBA Gu et al. backdoor watermarking of PLMs Chen et al. word2sentence backdoor
Closest papers by meaning
- On the Learnability of Watermarks for Language Models (ICLR 2024 · similarity 0.89)
- GaussMark: A Practical Approach for Structural Watermarking of Language Models (ICML 2025 · similarity 0.89)
- A Watermark for Order-Agnostic Language Models (ICLR 2025 · similarity 0.88)
- PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMs (NeurIPS 2026 · similarity 0.88)
- Watermarks in the Sand: Impossibility of Strong Watermarking for Language Models (ICML 2024 · similarity 0.87)
- Watermarking Makes Language Models Radioactive (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.