Research map / Large language models / Language, safety and interpretability
On the Empirical Power of Goodness-of-Fit Tests in Watermark Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates eight classical goodness-of-fit tests as watermark detectors across three unbiased watermarking schemes, three open LLMs, two datasets, and several temperatures and edit types, finding GoF tests improve power and robustness and benefit from repetition at low temperature.
Problem
Watermark detection for LLM text relies on ad hoc sum-based rules over pivotal statistics that are i.i.d. under human text, yet classical goodness-of-fit tests for this i.i.d. null remain largely unexplored in the watermarking setting, especially at low temperatures where detection is hard.
Approach
Apply general classical GoF tests (Phi, Kuiper, Kolmogorov-Smirnov, Anderson-Darling, Cramér-von Mises, Watson, Neyman smooth, chi-squared) to the p-values of pivotal statistics, comparing the empirical CDF to the known null CDF rather than summing scores; show that repetition at low temperature creates CDF deviations these tests exploit.
Key results
- On C4 at T=0.7 with n=400 under Gumbel-max, the best baseline has about 0.7% Type II error while several GoF tests reach about 0.2-0.3%.
- At T=0.3 and n=400 under Gumbel-max, the best baseline yields 15.1% Type II error versus 2.9% for Chi and 4.9% for And.
- Under information-rich edits at 50% for Inverse-transform, Kui reaches 0.4% Type II error versus 34.6% for the best baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a GoF test instead of the best sum-based baseline | Watermark detection with Gumbel-max, Inverse-transform, or SynthID · C4, T=0.3, n=400 | Type II error: 15.1 -> 2.9; 27.1 -> 3.6; 44.4 -> 10.4 | Helped |
| Use the Phi GoF test instead of the best sum-based baseline | Gumbel-max watermark detection · C4, T=0.7, n=200 and n=400 | Type II error: 0.6 -> 0.3; 0.7 -> 0.2 | Helped |
| Use distribution-level GoF detection instead of the best sum-based baseline | Watermark detection under 50% information-rich edits · C4, T=1.0 | Type II error: 38.0 -> 14.6; 34.6 -> 0.4; 94.5 -> 0.1 | Helped |
| Use GoF tests under word deletion and synonym substitution | Gumbel-max, Inverse-transform, and SynthID watermark detection · C4, T=1.0, common text edits | Type II error | Mixed |
| Remove repeated pivotal statistics (ablation) | GoF watermark detection · Low-temperature generated text | Type II error | Hurt |
| Use GoF tests rather than sum-based baselines when repetition is present | Watermark detection · Low-temperature settings | Type II error | Helped |
Surprising findings
- Text repetition at low temperature, which is usually considered harmful, gives GoF tests a unique advantage over sum-based baselines.
- Removing repeated pivotal statistics slightly degrades GoF performance, though it stays comparable to baselines.
Limitations
- Most GoF tests lack closed-form finite-sample null distributions, so critical values rely on asymptotics or Monte Carlo simulation.
- Green-red list watermarks were excluded because GoF tests reduce to counting green tokens.
- Repetition advantage may diminish as newer, more powerful LLMs produce less repetitive text.
Experimental setup
Datasets: C4 ELI5
Benchmarks: C4 ELI5
Metrics: Type I error Type II error FPR
Baselines: Aaronson's score (Ars) log score (Log) negative-sum score (Neg) identity-sum score (Sum) least-favorable test (Lst)
Builds on: Gumbel-max watermark SynthID least-favorable test Tr-GoF test
Closest papers by meaning
- On the Reliability of Watermarks for Large Language Models (ICLR 2024 · similarity 0.89)
- PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMs (NeurIPS 2026 · similarity 0.88)
- Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.88)
- GaussMark: A Practical Approach for Structural Watermarking of Language Models (ICML 2025 · similarity 0.87)
- Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach (NeurIPS 2025 · similarity 0.87)
- Provable Robust Watermarking for AI-Generated Text (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.