Research map / Image generation / Autoregressive and latent generation

Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models

CVPR 2025 · Image generation › Autoregressive and latent generation · approach: Reconstruction-based diffusion image detection · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FastProtect, which pre-trains a mixture-of-perturbations (with multi-layer VAE protection loss) plus adaptive targeted and adaptive-strength inference to protect images against diffusion mimicry at near-zero latency.

Problem

Existing image protection methods against diffusion-based mimicry (personalization such as LoRA/DreamBooth) optimize perturbations per image at inference, which is slow, and they trade off protection efficacy against invisibility.

Approach

Replace per-image iterative PGD with a pre-trained perturbation bank selected by VAE latent clustering (mixture-of-perturbations), add a multi-layer VAE feature protection loss during pre-training, and at inference pick a target by pattern-entropy matching and scale perturbation by an LPIPS-derived perceptual map.

Key results

• FastProtect runs in 2.9s on CPU and 0.04s on GPU for 512x512 images, about 125x and 175x faster than PhotoGuard, the second fastest method.
• On the Table 2 comparison at matched protection level, FastProtect achieves the best DISTS invisibility in most domains, e.g. 0.155 on objects versus 0.185-0.242 for other methods.
• Ablation: UAP drops FID from 227.6 (PhotoGuard) to 207.6; full MoP with MLP loss and adaptive targeting reaches 238.8.

Measured effects reported in the paper

Surprising findings

• Plain universal adversarial perturbation markedly reduces protection efficacy for mimicry prevention compared to per-image PGD.
• Using FastProtect as initialization for PGD refinement lets the baseline match results in 25 steps instead of 100 steps.

Limitations

• Still produces visible distortion inherent to adversarial perturbations.
• Protection efficacy measured mainly against LoRA on Stable Diffusion v1.5 with FID as the metric.

Experimental setup

Datasets: object, face, painting, and cartoon training and benchmark datasets (details in appendix)

Benchmarks: object, face, painting, cartoon domains

Metrics: FID DISTS latency

Baselines: AdvDM PhotoGuard Anti-DB Mist Impasto SDST (Diff-Protect)

Builds on: universal adversarial perturbation PhotoGuard Mist Impasto LoRA

Closest papers by meaning

• Toward effective protection against diffusion-based mimicry through score distillation (ICLR 2024 · similarity 0.91)
• DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing (ICLR 2025 · similarity 0.89)
• Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models (NeurIPS 2025 · similarity 0.89)
• Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion? (CVPR 2024 · similarity 0.89)
• Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency (ICML 2025 · similarity 0.89)
• Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI (ICLR 2025 · similarity 0.88)