Research map / Image generation / Autoregressive and latent generation
AquaLoRA: Toward White-box Protection for Customized Stable Diffusion Models via Watermark LoRA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AquaLoRA, a two-stage white-box watermarking method that embeds a 48-bit message into the Stable Diffusion U-Net via a LoRA with a message-dependent scaling matrix, trained with prior-preserving fine-tuning.
Problem
Protecting customized Stable Diffusion models against copyright misuse in a white-box setting, where adversaries have full model access and can remove or replace existing watermarking modules (post-hoc, VAE-based, or sampling-based watermarks).
Approach
Embed the watermark directly into the U-Net via a LoRA whose update includes a scaling matrix S = diag(1 + mapped secret bits), so messages can be changed without retraining; a prior-preserving fine-tuning loss targets the model's own noise predictions plus a fixed latent offset to minimize distribution shift.
Key results
- Average bit accuracy under distortions of 91.86% for AquaLoRA vs 84.19% for RivaGAN and 77.01% for Stable Signature; TPR under adversarial distortions 0.906.
- Fidelity DreamSim 0.201 with FID 24.88 vs 24.26 for no watermark; naive diffusion fine-tuning gives DreamSim 0.330 and 48.11% bit accuracy, while PPFT reaches 95.79%.
- Bit accuracy stays around 95% across samplers, CFG 5-10 and VAE decoders; coarse-type adaption raises average bit accuracy from 93.61% to 94.81%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AquaLoRA instead of Tree-ring watermarking | Stable Diffusion v1.5 · COCO2017 validation, adversarial distortions | TPR: 0.810 -> 0.906 | Helped |
| replace naive diffusion training with PPFT (ablation) | AquaLoRA · COCO2017 validation | Bit accuracy: 48.11 -> 95.79 | Helped |
| replace naive diffusion training with PPFT (ablation) | AquaLoRA · COCO2017 validation | DreamSim: 0.330 -> 0.201 | Helped |
| use orthogonal rather than normal mapper initialization (ablation) | AquaLoRA with PPFT · COCO2017 validation | Bit accuracy: 95.02 -> 95.79 | Helped |
| increase LoRA rank from 320 to 512 (ablation) | AquaLoRA with PPFT and orthogonal initialization · COCO2017 validation | Bit accuracy: 95.79 -> 96.29 | Helped |
| reduce LoRA rank from 320 to 128 (ablation) | AquaLoRA with PPFT and orthogonal initialization · COCO2017 validation | Bit accuracy: 95.79 -> 92.23 | Hurt |
| remove PRVL loss (ablation) | latent watermark pre-training · Latent watermark pre-training | PSNR: 29.85 -> 29.48 | Hurt |
| replace PRVL loss with MSE loss (ablation) | latent watermark pre-training · Latent watermark pre-training | PSNR: 29.85 -> 29.59 | Hurt |
| add coarse-type adaptation (ablation) | AquaLoRA on customized SD checkpoints · 25 customized SD checkpoints | Bit accuracy: 93.61% -> 94.81% | Helped |
| add coarse-type adaptation (ablation) | AquaLoRA on customized SD checkpoints · 25 customized SD checkpoints | DreamSim: 0.219 -> 0.204 | Helped |
Surprising findings
- Naive diffusion fine-tuning reaches only 48.11% bit accuracy while causing large output changes, showing that standard fine-tuning fails to embed the watermark without prior preservation.
- Merging two watermarked models at 0.5 ratio yields random extracted bits where the message bits disagree, a collusion behavior consistent with the marking assumption.
Limitations
- Weak against strong cropping and rotation due to the latent watermark pre-training stage's limited resilience.
- Does not handle editing, inpainting, or outpainting usage well; performance degrades at larger output image sizes.
- Cover-agnostic watermark can be estimated by averaging many latent vectors from in-distribution samples.
Experimental setup
Datasets: COCO2017 Stable-Diffusion-Prompts parti-prompts
Benchmarks: COCO2017 validation (FID) parti-prompts
Metrics: FID DreamSim bit accuracy TPR at FPR 1e-6 PSNR SSIM
Baselines: DwtDctSvd RivaGAN Stable Signature Tree-ring
Builds on: LoRA Tree-ring watermark Stable Signature RoSteALS DreamSim
Closest papers by meaning
- SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models (CVPR 2025 · similarity 0.90)
- Harnessing Frequency Spectrum Insights for Image Copyright Protection Against Diffusion Models (CVPR 2025 · similarity 0.90)
- Shallow Diffuse: Robust and Invisible Watermarking through Low-Dim Subspaces in Diffusion Models (NeurIPS 2025 · similarity 0.89)
- WMAdapter: Adding WaterMark Control to Latent Diffusion Models (ICML 2025 · similarity 0.89)
- Attack-Resilient Image Watermarking Using Stable Diffusion (NeurIPS 2024 · similarity 0.89)
- StableGuard: Towards Unified Copyright Protection and Tamper Localization in Latent Diffusion Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.