Research map / Large language models / Language, safety and interpretability

PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMs

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Text watermarking

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PRO, a learning-based text watermark for open-source LLMs that co-optimizes a trainable watermark mapping policy with the student LLM and adds a forgotten-perturbation-aware regularizer to resist merging and fine-tuning.

Problem

Watermarking open-source LLMs so that standard decoding by end users still yields detectable watermarked text, while surviving weight modifications such as fine-tuning and model merging. Existing distillation-based watermarks suffer from low learnability of predefined patterns (especially for n>1) a

Approach

Replace the predefined KGW/KTH watermark pattern with a trainable watermark mapping model (over BERT embeddings of n-gram prefixes) that is jointly optimized with the student LLM to produce learnable patterns, and detect with the learned mapping; add a forgotten-perturbation regularizer that simulates a red-token-driven weight update via adversarial anti-watermarked text and penalizes the resultin

Key results

• PRO reaches AUC 0.99 on unaltered LLaMA-3-8B while reducing perplexity by 20.5% relative to the best baseline across tested LLMs.
• After merging with the unwatermarked model at t=0.5 (SLERP), PRO retains AUC 0.876 versus 0.773 for the best KGW baseline (Gloaguen et al.).
• After 1500 fine-tuning steps on OpenMathInstruct, PRO keeps AUC 0.808 while Gu et al. KTH drops to 0.524.

Surprising findings

• Learning-based KGW watermarks with n=2 fail to be learned by the student (AUC 0.54) even with larger delta, motivating learnable mappings.
• Fine-tuning on anti-watermarked text causes a substantial drop in detectability, supporting red-token gradients as the main cause of watermark forgetting.

Limitations

• Robustness to merging and fine-tuning degrades at high merge ratios (AUC 0.68 at t=0.9) and long fine-tuning.
• Second-order FPL term is approximated as constant, which the paper does not justify rigorously.
• Evaluated on a limited set of models (LLaMA-3-8B, LLaMA-3.2-3B, Phi-2) and detection is mainly on C4 continuations.

Experimental setup

Datasets: OpenWebText C4 OpenMathInstruct OpenCodeInstruct

Benchmarks: C4 continuation detection (5,000 samples, 200 tokens) OpenMathInstruct fine-tuning robustness OpenCodeInstruct fine-tuning robustness

Metrics: AUC TPR at 5% FPR median perplexity

Baselines: Gu et al. KGW distillation Gu et al. KTH distillation Gloaguen et al. KGW (stable-parameter watermark)

Builds on: KGW KTH watermark Gloaguen et al. learning-based watermark distillation SLERP model merging SparseGPT

Closest papers by meaning

• An End-to-End Model for Logits-Based Large Language Models Watermarking (ICML 2025 · similarity 0.93)
• Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach (NeurIPS 2025 · similarity 0.92)
• Provable Robust Watermarking for AI-Generated Text (ICLR 2024 · similarity 0.92)
• Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.92)
• WaterMax: breaking the LLM watermark detectability-robustness-quality trade-off (NeurIPS 2024 · similarity 0.91)
• SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders (NeurIPS 2025 · similarity 0.91)