Research map / Image generation / Text-to-image diffusion
SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SleeperMark, a backdoor-based black-box watermark for text-to-image diffusion models that survives downstream fine-tuning (LoRA, DreamBooth, ControlNet) and VAE replacement.
Problem
Watermarks embedded in T2I diffusion models are forgotten when the model is fine-tuned for downstream tasks, and verification must work black-box via generated outputs only.
Approach
Two-stage method: pre-train a latent secret encoder/decoder for a 48-bit message, then fine-tune the diffusion backbone so that triggered prompts yield the watermark residual while regular prompts keep original behavior, using timestep-weighted single-step z0 losses to disentangle watermark from semantics.
Key results
- After LoRA style adaptation (rank 80, 2000 steps), SleeperMark keeps TPR@1e-6 FPR at 0.993 while AquaLoRA drops to 0.000.
- After DreamBooth fine-tuning for 1000 steps, SleeperMark retains TPR 0.934 versus 0.210 for WatermarkDM and 0.003 for AquaLoRA.
- On SD v1.4 model fidelity, SleeperMark reaches DreamSim 0.108 versus 0.176 for AquaLoRA and 0.279 for WatermarkDM, with 99.24% bit accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SleeperMark rather than AquaLoRA for latent watermarking | SD v1.4 · Stable Diffusion | DreamSim: 0.176 -> 0.108 | Helped |
| Use SleeperMark rather than AquaLoRA for latent watermarking | SD v1.4 · Stable Diffusion | Adversarial bit accuracy: 94.71 -> 97.98 | Helped |
| Use SleeperMark rather than AquaLoRA for pixel diffusion watermarking | DeepFloyd-IF · DeepFloyd | FID: 12.95 -> 13.03 | Hurt |
| Use SleeperMark rather than AquaLoRA for pixel diffusion watermarking | DeepFloyd-IF · DeepFloyd | DreamSim: 0.020 -> 0.018 | Helped |
| Use SleeperMark rather than AquaLoRA | SD v1.4 fine-tuned with LoRA rank 80 for 2000 steps · Naruto-style LoRA adaptation | TPR@10−6FPR: 0.000 -> 0.993 | Helped |
| Use SleeperMark rather than WatermarkDM | SD v1.4 fine-tuned with DreamBooth for 1000 steps · DreamBooth personalization | TPR@10−6FPR: 0.210 -> 0.934 | Helped |
| Use SleeperMark rather than WatermarkDM | SD v1.4 fine-tuned with ControlNet for 20000 steps · ControlNet Canny integration | TPR@10−6FPR: 0.000 -> 0.955 | Helped |
| Increase the trigger sequence length (ablation) | SleeperMark on SD v1.4 · LoRA rank 640 fine-tuning for 5000 steps | Bit accuracy after fine-tuning | Helped |
| Increase the trigger sequence length (ablation) | SleeperMark on SD v1.4 · Generated images after LoRA fine-tuning | DreamSim | Hurt |
| Fine-tune attention parameters in the up blocks only instead of down blocks (ablation) | SleeperMark on SD v1.4 · Regular prompts | Model fidelity | Helped |
Surprising findings
- Embedding the watermark in latent space makes extraction inherently robust to image distortions without a distortion layer during training.
- Watermark survives VAE decoder fine-tuning or replacement because extraction uses the original VAE encoder.
Limitations
- Cover-agnostic embedding leaves visible traces, making the watermark less concealed.
- Watermark effectiveness declines faster under full-weight fine-tuning such as DreamBooth.
- Slightly less resilient to Gaussian noise distortion.
Experimental setup
Datasets: COCO2014 Stable-Diffusion-Prompts Naruto dataset
Benchmarks: COCO2014
Metrics: FID CLIP score DreamSim bit accuracy TPR at FPR 1e-6
Baselines: DwtDctSvd Stable Signature WatermarkDM AquaLoRA
Builds on: AquaLoRA WaDiff StegaStamp Stable Signature WatermarkDM
Closest papers by meaning
- Harnessing Frequency Spectrum Insights for Image Copyright Protection Against Diffusion Models (CVPR 2025 · similarity 0.91)
- Black-Box Forgery Attacks on Semantic Watermarks for Diffusion Models (CVPR 2025 · similarity 0.90)
- T2SMark: Balancing Robustness and Diversity in Noise-as-Watermark for Diffusion Models (NeurIPS 2025 · similarity 0.90)
- AquaLoRA: Toward White-box Protection for Customized Stable Diffusion Models via Watermark LoRA (ICML 2024 · similarity 0.90)
- StableGuard: Towards Unified Copyright Protection and Tamper Localization in Latent Diffusion Models (NeurIPS 2025 · similarity 0.89)
- WMAdapter: Adding WaterMark Control to Latent Diffusion Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.