Research map / Large language models / Language, safety and interpretability
GaussMark: A Practical Approach for Structural Watermarking of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GaussMark, a structural LLM watermark that adds Gaussian noise to a single MLP weight matrix and detects it via a gradient-based Gaussian independence test with valid p-values, zero generation latency, and minimal quality loss.
Problem
Existing LLM watermarks that bias next-token sampling are often impractical due to generation latency, detection time, text quality degradation, or limited robustness; distortion-free constraints may be unnecessarily strict.
Approach
Watermark the model weights rather than the sampling process: perturb one MLP weight matrix with isotropic Gaussian noise at generation time, and detect by testing whether the noise direction is correlated with the gradient of the log-likelihood of the text, which yields an exactly standard-normal test statistic under the null.
Key results
- Watermarked Llama-3.1-8B, Mistral-7B and Phi-mini show SuperGLUE and GSM8K scores within confidence intervals of unwatermarked models (e.g., Llama GSM8K 0.6111 vs 0.6300 unwatermarked).
- Detection works with 1K-token generations at median p-values that decline with length; Llama-3.1-8B stays robust to token corruption with half tokens corrupted.
- Compared with Kirchenbauer-style schemes, GaussMark is less detectable and less robust but has much smaller quality impact and orders-of-magnitude faster generation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Gaussian noise to a single MLP weight | Llama 3.1-8B · GSM8K | accuracy: 0.6300 ± 0.0133 -> 0.6111 ± 0.0134 | No clear effect |
| Add Gaussian noise to a single MLP weight | Mistral-7B · SuperGLUE (Avg) | score: 0.6836 ± 0.0335 -> 0.7033 ± 0.0332 | No clear effect |
| Add Gaussian noise to a single MLP weight | Phi · GSM8K | accuracy: 0.8423 ± 0.0100 -> 0.8552 ± 0.0097 | No clear effect |
| Add Gaussian noise to a single MLP weight | Llama 3.1-8B · Alpaca Eval | win rate: 0.45; (0.46, 0.54) | Mixed |
| Restrict perturbations to lower principal components (ablation) | GaussMark · GSM8K | accuracy: 0.6300 ± 0.0133 -> 0.6133 ± 0.0134 (Llama); 0.4291 ± 0.0136 -> 0.4253 ± 0.0136 (Mistral); 0.8423 ± 0.0100 -> 0.8537 ± 0. | No clear effect |
| Restrict perturbations to lower principal components (ablation) | GaussMark · C4 | crossentropy: 1.9199 ± 0.0135 -> 1.9410 ± 0.0135 (Mistral); 2.1709 ± 0.0149 -> 2.1901 ± 0.0148 (Llama); 2.1396 ± 0.0133 -> 2.1620 ± 0. | No clear effect |
| Use rank-reduced watermarking to tolerate greater noise variance (ablation) | GaussMark · GSM8K | accuracy | Helped |
| Increase the number of generated tokens (ablation) | GaussMark · Watermark detection | detectability | Helped |
| Increase noise variance (ablation) | GaussMark · Watermark detection | median p-value | Mixed |
| Increase noise variance (ablation) | GaussMark · GSM8K | accuracy | Hurt |
| Apply token-level corruption to watermarked text (ablation) | GaussMark · Corrupted text | detectability | Hurt |
| Apply roundtrip translation through French (ablation) | GaussMark · Roundtrip-translated text | detectability | Hurt |
Surprising findings
- Perturbing the final softmax layer led to either negligible detectability or unacceptable quality loss, so only MLP block weights are perturbed.
- Larger noise variance eventually reduces detectability, attributed to the locality of the linear softmax approximation.
Limitations
- No formal guarantee of distortion-freeness or indistinguishability from unwatermarked text.
- Requires more tokens than some alternatives for reliable detection and needs white-box access to model weights for detection.
- Less robust to edit-level corruptions than some prior schemes, and rank-reduced variant is less robust.
Experimental setup
Datasets: C4 realnewslike
Benchmarks: SuperGLUE GSM8K AlpacaEval C4 cross-entropy
Metrics: p-value TPR at FPR 0.05 AUC accuracy win rate cross-entropy latency
Baselines: Kirchenbauer et al. (red-green list watermark) Christ et al. undetectable watermark SynthID-Text
Builds on: Kirchenbauer et al. green-red list watermark Aaronson-style watermarking undetectable watermarks (Christ et al.) model merging via linear interpolation low-rank weight modification
Closest papers by meaning
- BiMark: Unbiased Multilayer Watermarking for Large Language Models (ICML 2025 · similarity 0.91)
- On the Learnability of Watermarks for Language Models (ICLR 2024 · similarity 0.90)
- Black-Box Detection of Language Model Watermarks (ICLR 2025 · similarity 0.90)
- PRO: Enabling Precise and Robust Text Watermark for Open-Source LLMs (NeurIPS 2026 · similarity 0.90)
- Break the Trade-off Between Watermark Strength and Speculative Sampling Efficiency for Language Models (ICLR 2026 · similarity 0.90)
- Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.