Research map / Image generation / Text-to-image diffusion
Countering Personalized Text-to-Image Generation with Influence Watermarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InMark, a robust protective-noise (unlearnable example) method that embeds influence-guided watermarks in low-frequency DCT subspace to stop DreamBooth-style personalization of Stable Diffusion even after JPEG/WebP compression.
Problem
Protective noises that make personal reference images unlearnable for personalized text-to-image diffusion models fail once public platforms compress or modify the uploaded images.
Approach
Combines gradient descent restricted to a low-frequency DCT subspace with an influence-function (self-influence) direction, alternating between the two across epochs, so that the perturbed pixels most influential on the denoising loss survive compression and degrade personalized generation.
Key results
- Under JPEG compression, InMark achieves FDFR 0.67 on the 'photo' prompt versus 0.06 for ASPL-type baselines in clean conditions and keeps protection where LF-ASPL/LF-FSMG/LF-AdvDM fail (e.g., LF-ASPL avg FDFR 0.43 vs InMark 0.48 for portrait-averaged metrics).
- In the clean image-variation setting, InMark reaches PSNR 36.96 (avg) and RMSE 0.48 (avg) for the averaged metrics, the best image quality among protective methods in Table 1.
- InMark transfers to Textual Inversion and LoRA even though it is optimized with DreamBooth loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InMark instead of ASPL | DreamBooth on Stable Diffusion 2.1 · VGGFace2 image variations | BRISQUE (Avg.): 35.77 -> 43.51 | Helped |
| Use InMark instead of LF-FSMG | DreamBooth on Stable Diffusion 2.1 · VGGFace2 image variations | FDFR (Avg.): 0.56 -> 0.48 | Hurt |
| Use InMark instead of ASPL | DreamBooth on Stable Diffusion 2.1 · VGGFace2 image variations | ISM (Avg.): 0.28 -> 0.32 | Hurt |
| Add the influence-function update to the watermark optimization (ablation) | InMark · VGGFace2 and WikiArt personalized fine-tuning | accumulated loss | Helped |
| Restrict watermark optimization to the low-frequency subspace without the influence function (ablation) | LF-ASPL and LF-FSMG · VGGFace2 image variations | BRISQUE (Avg.): 43.51 vs. 28.30 | Hurt |
| Apply InMark to JPEG-compressed reference images | DreamBooth on Stable Diffusion 2.1 · VGGFace2 image variations | protective effectiveness after compression | Helped |
| Apply InMark to WebP-compressed reference images | DreamBooth on Stable Diffusion 2.1 · VGGFace2 image variations | protective effectiveness after compression | Helped |
Surprising findings
- High-frequency protective noises matter more than low-frequency ones for diffusion models, contrary to the CNN classifier finding that low-frequency restriction helps.
- Textual Inversion, which does not change model parameters, is protected by InMark with even stronger effect than DreamBooth.
Limitations
- Influence-function term needs Hessian computation, adding computational overhead.
- WebP compression yields only color distortion rather than full protection in some cases.
- Evaluated mainly on Stable Diffusion 2.1 with DreamBooth on a small number of reference images.
Experimental setup
Datasets: VGGFace2 WikiArt
Benchmarks: VGGFace2 WikiArt
Metrics: PSNR RMSE BRISQUE FDFR ISM
Baselines: ASPL FSMG AdvDM LF-ASPL LF-FSMG LF-AdvDM
Builds on: AdvDM ASPL DreamBooth unlearnable examples LF-PGD
Closest papers by meaning
- Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances (ICLR 2025 · similarity 0.87)
- Invisible Image Watermarks Are Provably Removable Using Generative AI (NeurIPS 2024 · similarity 0.85)
- Transferable Black-Box One-Shot Forging of Watermarks via Image Preference Models (NeurIPS 2025 · similarity 0.85)
- An Undetectable Watermark for Generative Image Models (ICLR 2025 · similarity 0.85)
- BitMark: Watermarking Bitwise Autoregressive Image Generative Models (NeurIPS 2025 · similarity 0.85)
- On the Robustness of Watermarking for Autoregressive Image Generation (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.