Research map / Image generation / Text-to-image diffusion
PID: Prompt-Independent Data Protection Against Latent Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PID, a prompt-independent data protection method that perturbs the LDM's VAE visual encoder latent mean and log-variance so protection holds even when protector and exploiter prompts differ.
Problem
Existing defenses against few-shot personalization of latent diffusion models assume the data protector and the malicious exploiter use identical textual prompts; this breaks down under prompt mismatch, weakening protection.
Approach
Instead of attacking the text-conditioned training loss, perturb pixels so that the VAE encoder's prompt-independent latent distribution (mean and log-variance) shifts strongly, using a loss L_add-log = L_mean + log variance ratio; it can also be combined with ASPL/FSGM.
Key results
- Under prompt mismatch on CelebA-HQ with SD v1.5 (frozen text encoder), PID reduces FDS to 0.254 versus 0.370 for ASPL and 0.374 for FSGM, with FID 352.8.
- PID uses roughly 5G GPU memory versus 24G for UNet-based defenses, while matching or beating them across the four training configurations.
- Combining PID with ASPL (ASPL+PID) lowers FDS from 0.370 to 0.254 under prompt mismatch on SD v1.5 (frozen text encoder).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the matched protection/fine-tuning prompt with a mismatched prompt | FSGM · CelebA-HQ, SD v1.5, unfrozen text encoder | FDS: 0.277 -> 0.387 | Hurt |
| use PID instead of FSGM | SD v1.5 data protection · CelebA-HQ, prompt mismatch, frozen text encoder | FDS: 0.374 -> 0.254 | Helped |
| use PID instead of FSGM | SD v1.5 data protection · CelebA-HQ, prompt mismatch, frozen text encoder | FID: 224.460 -> 352.795 | Helped |
| add PID to ASPL | ASPL · CelebA-HQ, prompt mismatch, frozen text encoder | FDS: 0.370 -> 0.254 | Helped |
| add PID to ASPL | ASPL · CelebA-HQ, prompt mismatch, frozen text encoder | FID: 271.893 -> 352.795 | Helped |
| add PID to FSGM | FSGM · CelebA-HQ, prompt mismatch, frozen text encoder | FDS: 0.374 -> 0.276 | Helped |
| replace the combined mean and log-variance target with a targeted mean loss (ablation) | PID loss design · CelebA-HQ, fine-tuning on protected data | FID: 271.540 -> 411.990 | Helped |
| replace the combined mean and log-variance target with the sampled latent loss (ablation) | PID loss design · CelebA-HQ, fine-tuning on protected data | FID: 265.588 -> 411.990 | Helped |
| replace the combined mean and log-variance target with the unlogged sum of mean and variance losses (ablation) | PID loss design · CelebA-HQ, fine-tuning on protected data | FID: 268.260 -> 411.990 | Helped |
| maximize the combined mean and log-variance target instead of mean difference only (ablation) | visual-encoder data protection · CelebA-HQ, fine-tuning on protected data | FID: 243.292 -> 411.990 | Helped |
| maximize the combined mean and log-variance target instead of variance difference only (ablation) | visual-encoder data protection · CelebA-HQ, fine-tuning on protected data | FID: 265.337 -> 411.990 | Helped |
| increase prompt aggregation from one prompt to four prompts (ablation) | ASPL prompt aggregation · CelebA-HQ, prompt mismatch, SD v1.5, frozen text encoder | FDS: 0.385 -> 0.401 | Hurt |
Surprising findings
- Adaptive attacks that fix or clip the encoder variance to near-normal values still leave FDS below 0.3, so PID stays effective.
- Naively aggregating ASPL over several prompts (ASPL-k) does not substantially fix prompt-mismatch degradation.
Limitations
- PID's protection degrades substantially under JPEG compression (FDS 0.345 for compression vs about 0.21 for other corruptions).
- PID is not sufficiently effective against zero-shot image inpainting editing in the authors' test.
- The trade-off coefficient lambda=0.05 was not exhaustively searched; combining PID with FSGM did not improve image quality.
Experimental setup
Datasets: CelebA-HQ VGGFACE VGGFace2
Benchmarks: CelebA-HQ VGGFACE
Metrics: FDS FID IQS BRISQUE SSIM PSNR LPIPS
Baselines: AdvDM FSGM ASPL Mist
Builds on: AdvDM ASPL FSGM DreamBooth PGD
Closest papers by meaning
- Anti-Diffusion: Preventing Abuse of Modifications of Diffusion-Based Models (AAAI 2025 · similarity 0.89)
- Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models (NeurIPS 2025 · similarity 0.88)
- Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models (NeurIPS 2024 · similarity 0.88)
- Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion? (CVPR 2024 · similarity 0.88)
- RAPID: Retrieval Augmented Training of Differentially Private Diffusion Models (ICLR 2025 · similarity 0.87)
- Enhancing Facial Privacy Protection via Weakening Diffusion Purification (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.