Research map / Large language models / Language, safety and interpretability

Invisible Ink, Visible Lies: How Production Watermarking Causes LLMs to Hallucinate

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Task vectors and steering · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that decoding-time text watermarks cause factual errors in a controlled RAG setting, attributes them to token-level perturbations and prefix-induced attention drift, and proposes two plug-in interventions (FPTI, FPAI) that cut factual degradation by about 90% at matched TPR.

Problem

Text watermarking is evaluated for detectability, robustness and fluency, but its effect on factual reliability when the needed evidence is in context is largely unknown. Watermarks can flip correct answers to incorrect ones even when an unwatermarked model answers correctly.

Approach

Defines watermarking hallucination as a paired factual accuracy decrease (Delta_fact) between unwatermarked and watermarked outputs under the same RAG context, decomposes its cause into current-step token perturbation and prefix-induced attention drift, and introduces FPTI (exempting fact-bearing tokens from watermark perturbation) and FPAI (adding attention logit compensation on fact-bearing cont

Key results

• Under KGW on LLaMA3.1-8B at TPR@1%FPR=0.90, Delta_fact is 12.9%; FPTI reduces it to 2.4% and FPTI+FPAI to 0.9%, a 93.0% reduction over KGW.
• Across six watermarks and three models, FPTI+FPAI lowers Delta_fact at TPR=0.90 to roughly 0.4-1.3% from 6.5-15.7% for the unmodified watermarks.
• DeCoRe reduces KGW Delta_fact to 3.2% at TPR=0.90 but with a much lower decoding speed (1.10 vs 1.48 tokens/s).

Surprising findings

• Factual tokens lie largely above SWEET's entropy threshold, so entropy-based protection does not cover many fact-bearing decisions.
• Watermarked outputs can have lower perplexity yet contain entity-level factual errors, so fluency metrics miss these errors.

Limitations

• Interventions do not eliminate factual degradation under strong watermarking or long generations, with residual error growing with generation depth.
• Evaluation is limited to a controlled RAG setting with four fact types and may not generalize to open-ended long-form generation.
• Factuality labels rely on rule-based matching with manual audit on samples rather than human-validated full annotation.

Experimental setup

Datasets: Natural Questions Wikipedia passages (RAG contexts)

Benchmarks: controlled RAG factuality set (numerical, identifier, temporal, qualitative facts)

Metrics: Delta_fact (net factual accuracy decrease) TPR@1%FPR perplexity decoding speed

Baselines: KGW SWEET DiPmark GumbelSoft SynthID-style watermarking Gumbel-Max watermarking DoLa DeCoRe

Builds on: KGW SynthID DoLa DeCoRe Natural Questions

Closest papers by meaning

• Unbiased Watermark for Large Language Models (ICLR 2024 · similarity 0.86)
• Break the Trade-off Between Watermark Strength and Speculative Sampling Efficiency for Language Models (ICLR 2026 · similarity 0.86)
• On the Reliability of Watermarks for Large Language Models (ICLR 2024 · similarity 0.86)
• Discovering Spoofing Attempts on Language Model Watermarks (ICML 2025 · similarity 0.86)
• Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data? (AAAI 2025 · similarity 0.85)
• Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.85)