Research map / Image generation / Text-to-image diffusion
SimAC: A Simple Anti-Customization Method for Protecting Face Privacy against Text-to-Image Synthesis of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SimAC, an add-on to adversarial anti-customization methods (Anti-DreamBooth, AdvDM) that uses adaptive greedy timestep selection and a feature interference loss on U-Net decoder features to better disrupt identity in DreamBooth-based face customization.
Problem
Existing adversarial anti-customization methods for diffusion models optimize the training loss over randomly sampled timesteps, where gradients vanish at large timesteps, making noise optimization ineffective and leaving face identity recoverable from protected images.
Approach
Select optimization timesteps adaptively via greedy interval pruning based on gradient magnitude (focusing on small timesteps where high-frequency information matters), and add a feature interference loss that pushes intermediate U-Net decoder features (layers 9-11) away from those of clean images, combined with the vanilla diffusion loss.
Key results
- On CelebA-HQ with prompt 'a dslr portrait of sks person', Anti-DB + SimAC reduces ISM from 0.19 to 0.12 and raises FDFR from 86.80 to 96.87 over Anti-DreamBooth alone.
- On VGG-Face2 with prompt 'a photo of sks person looking at the mirror', Anti-DB + SimAC raises FDFR from 45.00 to 89.60 versus Anti-DreamBooth.
- Under prompt mismatch (identifier 'sks' changed to 't@t') and model mismatch (SD v1.4 vs v2.1), protection degrades but ISM stays relatively low, per the authors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SimAC to Anti-DB | Anti-DB · CelebA-HQ, a photo of sks person | ISM: 0.28 -> 0.31 | Hurt |
| Add SimAC to Anti-DB | Anti-DB · CelebA-HQ, a photo of sks person | FDFR: 77.28 -> 87.07 | Helped |
| Add SimAC to Anti-DB | Anti-DB · CelebA-HQ, a photo of sks person | BRISQUE: 37.43 -> 38.86 | Helped |
| Add SimAC to Anti-DB | Anti-DB · CelebA-HQ, a photo of sks person | SER-FQA: 0.20 -> 0.21 | Hurt |
| Add SimAC to Anti-DB | Anti-DB · VGG-Face2, a photo of sks person | ISM: 0.30 -> 0.31 | Hurt |
| Add SimAC to Anti-DB | Anti-DB · VGG-Face2, a photo of sks person | FDFR: 64.67 -> 80.27 | Helped |
| Add SimAC to Anti-DB | Anti-DB · VGG-Face2, a photo of sks person | BRISQUE: 37.89 -> 40.71 | Helped |
| Add SimAC to Anti-DB | Anti-DB · VGG-Face2, a photo of sks person | SER-FQA: 0.22 -> 0.22 | No clear effect |
Surprising findings
- Gradients of perturbed images become nearly zero at large diffusion timesteps, so a large share of random-timestep optimization steps are ineffective.
- In one model-mismatch setting (noise from SD v2.1, customization with SD v1.4), BRISQUE and FDFR shift sharply, e.g. FDFR 99.86 with BRISQUE 62.09 for the mismatched condition.
Limitations
- Evaluated only on two face datasets with about 50 identities each.
- Black-box robustness degrades under prompt and model mismatch, though ISM stays relatively stable.
- Improvements are measured by FDFR and identity metrics on DreamBooth fine-tuning; generalization to other customization methods is only lightly tested.
Experimental setup
Datasets: CelebA-HQ VGGFace2
Benchmarks: CelebA-HQ VGGFace2
Metrics: ISM (ArcFace cosine similarity) FDFR (face detection failure rate, RetinaFace) BRISQUE SER-FIQ
Baselines: PhotoGuard AdvDM Anti-DreamBooth
Builds on: Anti-DreamBooth AdvDM PGD DreamBooth
Closest papers by meaning
- Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models (NeurIPS 2025 · similarity 0.90)
- Harnessing Global-Local Collaborative Adversarial Perturbation for Anti-Customization (CVPR 2025 · similarity 0.90)
- Targeted Attack Improves Protection against Unauthorized Diffusion Customization (ICLR 2025 · similarity 0.89)
- Enhancing Facial Privacy Protection via Weakening Diffusion Purification (CVPR 2025 · similarity 0.88)
- Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models (NeurIPS 2024 · similarity 0.87)
- DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.