Research map / Adversarial robustness and security / Attacks on text and graphs
WAVES: Benchmarking the Robustness of Image Watermarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WAVES, a benchmark stress-testing image watermark robustness with 26 attacks (distortions, regeneration, adversarial) across Stable Signature, Tree-Ring, and StegaStamp, scored by TPR@0.1%FPR against normalized image quality.
Problem
Existing image watermark robustness evaluations are inconsistent in quality measures, statistical thresholds, and attack types, giving an incomplete picture of watermark vulnerabilities for AI-generated images.
Approach
A standardized benchmark that jointly plots detection/identification performance (TPR@0.1%FPR, identification accuracy) against an aggregated normalized image-quality degradation score, with new attacks such as rinsing regeneration, grey/black-box adversarial embedding attacks, and surrogate-detector adversarial attacks.
Key results
- Tree-Ring is highly vulnerable to grey-box embedding attack AdvEmbG-KLVAE8, with TPR@0.1%FPR dropping to near zero (rank 3 leaderboard).
- Regen-Diff reduces Stable Signature TPR@0.1%FPR to about 0.001 average, with StegaStamp mildly affected only by diffusive regenerations.
- StegaStamp ranks as the most robust of the three watermarks across attack categories, while Stable Signature is least robust.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the victim KL-VAE (f8) encoder for embedding attacks | Tree-Ring · WAVES detection | TPR@0.1%FPR: Avg P 0.448 | Hurt |
| Use a surrogate detector trained on watermarked and non-watermarked victim-generator images | Tree-Ring · WAVES detection | TPR@0.1%FPR: Avg P 0.499 | Hurt |
| Use a surrogate message classifier trained on watermarked images from two users | Tree-Ring · WAVES detection | TPR@0.1%FPR: Avg P 0.492 | Hurt |
| Train a surrogate detector using real images and watermarked images | Tree-Ring · WAVES detection | TPR@0.1%FPR: Avg P 1.000 | No clear effect |
| Regenerate images once with diffusion | Stable Signature · WAVES detection | TPR@0.1%FPR: Avg P 0.001 | Hurt |
| Apply four repeated diffusion regeneration cycles | Stable Signature · WAVES detection | TPR@0.1%FPR: Avg P 0.000 | Hurt |
| Apply rotation distortion | StegaStamp · WAVES detection | TPR@0.1%FPR: Avg P 0.357 | Hurt |
| Regenerate images once with prompted diffusion | StegaStamp · WAVES detection | TPR@0.1%FPR: Avg P 0.940 | No clear effect |
| Train Stable Signature's extractor with blur and rotation transformations as data augmentations | Stable Signature · Blur and rotation transformations | robustness | Helped |
Surprising findings
- Tree-Ring is not robust to regeneration attacks, contrary to prior claims, with a single regeneration significantly harming TPR while keeping reasonable CLIP-FID.
- Adversarial attacks on surrogate detectors fail to transfer to Stable Signature and StegaStamp, and the Real&WM surrogate attack fails entirely on Tree-Ring.
Limitations
- Attacks are mostly universal and do not exploit watermark-specific knowledge, so effectiveness may be underestimated.
- Surrogate detector attacks often fail to transfer due to architecture mismatch with the true detector.
- Only three watermark methods are evaluated in depth; other watermarks are not benchmarked in the main results.
Experimental setup
Datasets: DiffusionDB MS-COCO DALL-E 3 dataset
Benchmarks: WAVES
Metrics: TPR@0.1%FPR identification accuracy normalized quality degradation PSNR SSIM NMI FID CLIP-FID LPIPS aesthetics and artifacts scores
Baselines: Stable Signature Tree-Ring StegaStamp
Builds on: Tree-Ring Stable Signature StegaStamp PGD
Closest papers by meaning
- ROBIN: Robust and Invisible Watermarks for Diffusion Models with Adversarial Optimization (NeurIPS 2024 · similarity 0.87)
- Robustness of AI-Image Detectors: Fundamental Limits and Practical Attacks (ICLR 2024 · similarity 0.86)
- Hidden in the Noise: Two-Stage Robust Watermarking for Images (ICLR 2025 · similarity 0.86)
- Attack-Resilient Image Watermarking Using Stable Diffusion (NeurIPS 2024 · similarity 0.85)
- An Undetectable Watermark for Generative Image Models (ICLR 2025 · similarity 0.85)
- Transferable Black-Box One-Shot Forging of Watermarks via Image Preference Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.