Research map / Large language models / Language, safety and interpretability
Bileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signature
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bileve, a bi-level signature watermark for LLMs that pairs digital-signature bits for integrity checks with a rank-based statistical watermark to resist spoofing while staying detectable under edits.
Problem
LLM text watermarks are robust to removal but vulnerable to spoofing (symmetric key abuse, learnability, and semantic manipulation that keeps the watermark while changing meaning), which can wrongly blame the model developer.
Approach
Embed content-dependent digital signature bits (verifiable with a public key) into tokens via rejection-style sampling, while a coarse-grained rank-based sampling watermark (WRA score with shift-generate) keeps a global statistical signal robust to edits; combining the two levels lets the detector distinguish five cases including tampering evidence.
Key results
- With 10% of tokens edited, Bileve keeps F1 of 0.999 and FPR of 0 while SLS's TPR and FPR both drop to 0.
- Rank-based sampling with shift-generate reduces perplexity by 23.08% on OpenGen with OPT-1.3B versus SLS.
- Under semantic manipulation, Unigram retains z-score above 6 on attacked text, while Bileve's local/global tests flag tampering via a p-value of 0.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace a few tokens to manipulate semantics | Unigram · Semantic manipulation | z-score: 10.9 -> 10.5 | No clear effect |
| Use Bileve's coarse-grained alignment signal alongside signature verification | Bileve · Semantic manipulation | provenance detection: p-value: 0.01 | Helped |
| Use a bi-level signature with coarse-grained signal | Bileve · Generated text with 10% token edits | F1 score: 0.999 | Helped |
| Use a bi-level signature with coarse-grained signal | Bileve · Generated text with 10% token edits | FPR: 0 | Helped |
| Replace nucleus sampling with rank-based sampling and shift-generate | SLS · OpenGen with OPT-1.3B | perplexity: 23.08% perplexity reduction | Helped |
| Embed signature bits precisely into selected tokens | SLS and Bileve · Text generation | perplexity | Hurt |
| Use rank-based sampling instead of nucleus sampling | SLS · Text generation | perplexity | Helped |
Surprising findings
- Unigram achieves lower perplexity than human text largely because it produces repetitive generations, so perplexity is a misleading quality measure here.
- Semantic manipulation with a few word replacements keeps Unigram's watermark detectable (z-score 10.5 vs 10.9), exposing spoofing via robustness.
Limitations
- Signature embedding increases perplexity relative to Unigram, with the authors noting precise bit matching compromises fluency.
- Efficiency and generation quality are acknowledged as needing improvement; signature bits are embedded per token, which is costly for low-entropy text.
- Evaluation limited to OPT-1.3B and LLaMA-7B and a small set of tasks and attacks.
Experimental setup
Datasets: OpenGen (WikiText-103 derived) LFQA
Benchmarks: OpenGen LFQA
Metrics: TPR FPR F1 perplexity p-value z-score
Baselines: Unigram watermark SLS (single-level signature)
Builds on: Unigram watermark Christ et al. undetectable watermark (shift-generate) SLS digital signature watermark nucleus sampling
Closest papers by meaning
- Discovering Spoofing Attempts on Language Model Watermarks (ICML 2025 · similarity 0.89)
- BiMark: Unbiased Multilayer Watermarking for Large Language Models (ICML 2025 · similarity 0.88)
- Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data? (AAAI 2025 · similarity 0.87)
- StealthInk: A Multi-bit and Stealthy Watermark for Large Language Models (ICML 2025 · similarity 0.87)
- Enhancing LLM Watermark Resilience Against Both Scrubbing and Spoofing Attacks (NeurIPS 2025 · similarity 0.86)
- Watermark Stealing in Large Language Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.