Research map / Image generation / Autoregressive and latent generation
Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes APDM, which protects subjects against DreamBooth-style personalization by directly updating diffusion model weights (DPO-style protective loss plus a dual-path L2P optimization) instead of perturbing images.
Problem
Personalized text-to-image diffusion enables unauthorized generation of specific subjects (faces, objects); existing data-poisoning protections require perturbing all user images and fail when clean images are mixed in or simple transforms are applied.
Approach
Shift protection from images to the diffusion model parameters: a Diffusion-DPO-style protective loss pushes the model away from target-subject images toward generic images, and L2P simulates inner personalization steps so the protection gradient accounts for future personalization trajectories.
Key results
- APDM reaches average DINO 0.1167 (lower is better protection) and BRISQUE 50.50 versus 0.4411 and 27.69 for the best data-poisoning baseline (SimAC, all-perturbed).
- Under the one-perturbed-image scenario, baselines remain ineffective while APDM with clean images keeps DINO around 0.14 for person and 0.10 for dog.
- Preservation: APDM FID 28.85 vs 25.98 for Stable Diffusion; CLIP 0.2853 vs 0.2878; GenEval 0.4017 vs 0.4303.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use APDM rather than the strongest data-poisoning baseline | Stable Diffusion 1.5 · Person protection | DINO: 0.4448 -> 0.1375 | Helped |
| use APDM rather than the strongest data-poisoning baseline | Stable Diffusion 1.5 · Dog protection | DINO: 0.4106 -> 0.0959 | Helped |
| use APDM rather than the strongest data-poisoning baseline | Stable Diffusion 1.5 · Person protection | BRISQUE: 26.90 -> 40.25 | Helped |
| use APDM rather than the strongest data-poisoning baseline | Stable Diffusion 1.5 · Dog protection | BRISQUE: 31.64 -> 60.74 | Helped |
| protect the model with APDM instead of the original model | Stable Diffusion · COCO 2014 val | FID: 25.98 -> 28.85 | Hurt |
| protect the model with APDM instead of the original model | Stable Diffusion · COCO 2014 val | CLIP score: 0.2878 -> 0.2853 | Hurt |
| protect the model with APDM instead of the original model | Stable Diffusion · COCO 2014 val | TIFA: 78.76 -> 75.91 | Hurt |
| protect the model with APDM instead of the original model | Stable Diffusion · COCO 2014 val | GenEval: 0.4303 -> 0.4017 | Hurt |
| add positive-negative image pairing (ablation) | APDM · Person protection | DINO: 0.2770 -> 0.1375 | Helped |
| add L2P personalization-trajectory optimization (ablation) | APDM · Person protection | DINO: 0.4454 -> 0.1375 | Helped |
| increase beta from 1 to 10 (ablation) | APDM · Person protection | DINO: 0.1375 -> 0.5392 | Hurt |
| increase N_per from 5 to 20 (ablation) | APDM with L2P · Person protection | DINO: 0.3371 -> 0.1375 | Helped |
Surprising findings
- The naive adversarial loss (negated personalization loss plus prior preservation) makes FID rise over training, and the paper argues it provably fails to converge.
- Ablating L2P's personalization path (N_per) monotonically improves protection, with no saturation observed up to 20.
Limitations
- Preservation of generation quality is slightly reduced (FID 28.85 vs 25.98, GenEval 0.4017 vs 0.4303).
- Evaluated on a limited set of subjects (persons, dogs, cats, sneakers, glasses, clocks) with few images each.
- Protection requires per-model optimization (about 9 GPU hours), and the paper's theoretical convergence argument relies on first-order approximations.
Experimental setup
Datasets: DreamBooth dataset Anti-DreamBooth dataset CelebA-HQ VGGFace2 MS-COCO 2014
Benchmarks: DreamBooth dataset Anti-DreamBooth dataset MS-COCO 2014 validation
Metrics: DINO score BRISQUE FID CLIP score TIFA GenEval
Baselines: DreamBooth AdvDM Anti-DreamBooth SimAC PAP Custom Diffusion
Builds on: DreamBooth Diffusion-DPO Anti-DreamBooth PGD
Closest papers by meaning
- Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion? (CVPR 2024 · similarity 0.91)
- Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models (NeurIPS 2024 · similarity 0.90)
- SimAC: A Simple Anti-Customization Method for Protecting Face Privacy against Text-to-Image Synthesis of Diffusion Models (CVPR 2024 · similarity 0.90)
- Targeted Attack Improves Protection against Unauthorized Diffusion Customization (ICLR 2025 · similarity 0.90)
- Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models (CVPR 2025 · similarity 0.89)
- Enhancing Facial Privacy Protection via Weakening Diffusion Purification (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.