Research map / Large language models / Language, safety and interpretability
A Watermark for Order-Agnostic Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PATTERN-MARK, a statistical watermark for order-agnostic language models (ProteinMPNN, CMLM) that embeds a Markov-chain-generated key sequence and detects it via counts of high-frequency key patterns with a dynamic-programming false-positive-rate test.
Problem
Statistical watermarks rely on previously generated tokens as context, which order-agnostic LMs (non-sequential decoders) do not provide consistently, so existing schemes cannot be applied to them for detection.
Approach
Generate a Markov-chain key sequence with high-frequency alternating key patterns, seed the per-position token distribution with each key, recover the key sequence from the text at detection, and run a hypothesis test on pattern counts with an exact DP-computed null distribution.
Key results
- On protein generation at delta=1.50, PATTERN-MARK reaches 99.87% TPR at 0.1% FPR versus 85.14% for Unigram and 0.80% for Soft watermark, with pLDDT 84.02 vs 85.46 unwatermarked.
- On machine translation at delta=5, PATTERN-MARK achieves 98.01% TPR at 0.1% FPR versus 95.41% for Unigram, but BLEU drops to 24.23 vs 22.23 for Unigram and 32.85 unwatermarked.
- Under ChatGPT paraphrase attack at delta=5 on translation, PATTERN-MARK keeps 89.53% TPR at strength 0.3 versus 64.01% for Unigram.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PATTERN-MARK (δ=1.50) instead of Unigram (δ=1.50) | ProteinMPNN watermarking · Protein generation | TPR@FPR 0.1%: 85.14 -> 99.87 | Helped |
| Use PATTERN-MARK (δ=1.50) instead of no watermark | ProteinMPNN generation · Protein generation | pLDDT: 85.46 -> 84.02 | Hurt |
| Use PATTERN-MARK (δ=1.25) instead of Unigram (δ=1.50) | ProteinMPNN watermarking · Protein generation | TPR@FPR 0.1%: 85.14 -> 96.12 | Helped |
| Use PATTERN-MARK (δ=5) instead of Unigram (δ=5) | CMLM watermarking · Machine translation | TPR@FPR 0.1%: 95.41 -> 98.01 | Helped |
| Use PATTERN-MARK (δ=5) instead of no watermark | CMLM generation · Machine translation | BLEU: 32.85 -> 24.23 | Hurt |
| Use pattern length m=5 for protein generation (ablation) | PATTERN-MARK · Protein generation | detection efficiency | Helped |
| Use pattern length m=4 for machine translation (ablation) | PATTERN-MARK · Machine translation | detection efficiency | Helped |
| Increase pattern length beyond its optimal value (ablation) | PATTERN-MARK · Protein generation and machine translation | detection efficiency | Hurt |
| Vary transition-matrix probability a11 (ablation) | PATTERN-MARK · Protein generation and machine translation | generation quality | No clear effect |
| Set transition-matrix probability a11=0 (ablation) | PATTERN-MARK · Protein generation and machine translation | watermark detectability | Helped |
Surprising findings
- At matched delta, PATTERN-MARK's translation BLEU at delta=5 (24.23) is lower than Unigram's (22.23) is not the case; PATTERN-MARK at delta=5 has higher BLEU than Soft and Unigram only at lower delta, so quality-detectability comparisons depend on delta alignment.
- Fixing the transition matrix to strongly alternating keys (a11=0) was chosen as it did not hurt generation quality across the tested range.
Limitations
- Detection requires a pattern-count null distribution and quadratic-time DP, and the optimized version is restricted to two keys with alternating patterns.
- Robustness evaluated only with random token modification (protein) and ChatGPT paraphrasing (translation); paraphrase attacks not tested on protein generation.
- Evaluated on only two order-agnostic models and two tasks with a small number of samples (about 750-1000).
Experimental setup
Datasets: PDB structures (2020-2023, 747 filtered samples) WMT16-style news crawl news.2015.ro (1003 filtered samples)
Benchmarks: Protein Data Bank protein design (pLDDT, protein diversity) Romanian-English machine translation (BLEU)
Metrics: TPR@FPR pLDDT protein diversity (n-gram entropy) BLEU
Baselines: Soft watermark Multikey watermark Unigram watermark
Builds on: Soft watermark (red-green list) Unigram watermark Multikey watermark
Closest papers by meaning
- GaussMark: A Practical Approach for Structural Watermarking of Language Models (ICML 2025 · similarity 0.88)
- Task-Agnostic Language Model Watermarking via High Entropy Passthrough Layers (AAAI 2025 · similarity 0.88)
- Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.88)
- Break the Trade-off Between Watermark Strength and Speculative Sampling Efficiency for Language Models (ICLR 2026 · similarity 0.87)
- Watermarks in the Sand: Impossibility of Strong Watermarking for Language Models (ICML 2024 · similarity 0.86)
- Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.