Research map / Image generation / Editing and style transfer
Distraction is All You Need: Memory-Efficient Image Immunization against Diffusion-Based Image Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a semantic attack that suppresses cross-attention on protected content in T2I diffusion models, plus timestep universal gradient updating, to immunize images against diffusion-based editing and inpainting at roughly half the GPU memory of PhotoGuard's diffusion attack.
Problem
Images can be maliciously edited with text-to-image diffusion models; existing immunization (PhotoGuard) backpropagates through the full diffusion process, using large GPU memory and performing unstably under inpainting masks.
Approach
Attack the intermediate cross-attention maps for the concept to protect (e.g., "woman") rather than the final output, so gradients come from single denoising U-Net passes at sampled timesteps; averaging gradients across timesteps yields a timestep-universal perturbation.
Key results
- On Stable Diffusion V1.4, the semantic attack gives SSIM 0.4470 vs 0.6504 for the diffusion attack and LPIPS 0.5901 vs 0.4056 (higher is more disruption).
- GPU memory of the semantic attack barely grows with the number of diffusion timesteps, while the diffusion attack's memory scales with timestep count, reaching roughly half the memory of the prior method.
- The attack also disrupts null-text inversion editing and EDICT editing of immunized images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use semantic attack instead of the strongest baseline (Diffusion Attack) | Image immunization · Image editing, Stable Diffusion V1.4 | PSNR: 18.2617 -> 15.1487 | Helped |
| Use semantic attack instead of the strongest baseline (Encoder Attack) | Image immunization · Image editing, Stable Diffusion V1.4 | SSIM: 0.6318 -> 0.4470 | Helped |
| Use semantic attack instead of the strongest baseline (Encoder Attack) | Image immunization · Image editing, Stable Diffusion V1.4 | VIFp: 0.2118 -> 0.1462 | Helped |
| Use semantic attack instead of the strongest baseline (Diffusion Attack) | Image immunization · Image editing, Stable Diffusion V1.4 | FSIM: 0.7693 -> 0.6584 | Helped |
| Use semantic attack instead of the strongest baseline (Diffusion Attack) | Image immunization · Image editing, Stable Diffusion V1.4 | LPIPS: 0.4056 -> 0.5901 | Helped |
| Use semantic attack instead of the strongest baseline (Encoder Attack) | Image immunization · Image editing, Stable Diffusion V2.0 | PSNR: 18.5955 -> 18.0589 | Helped |
| Use semantic attack instead of the strongest baseline (Encoder Attack) | Image immunization · Image editing, Stable Diffusion V2.0 | SSIM: 0.6045 -> 0.4719 | Helped |
| Use semantic attack instead of the strongest baseline (Encoder Attack) | Image immunization · Image editing, Stable Diffusion V2.0 | VIFp: 0.1618 -> 0.1008 | Helped |
| Use semantic attack instead of the strongest baseline (Encoder Attack) | Image immunization · Image editing, Stable Diffusion V2.0 | FSIM: 0.7453 -> 0.7313 | Helped |
| Use semantic attack instead of the strongest baseline (Encoder Attack) | Image immunization · Image editing, Stable Diffusion V2.0 | LPIPS: 0.5799 -> 0.6019 | Helped |
Surprising findings
- On Stable Diffusion V1.4 the semantic attack's PSNR (15.15) is lower than the diffusion attack's (18.26), while its perceptual metrics are better.
Limitations
- Ineffective after noise purification such as blurring or JPEG compression, as the authors acknowledge.
- Evaluation uses a small custom set of 150 generated images with no public benchmark.
Experimental setup
Datasets: 150 images generated by diffusion model featuring 3 objects with editing prompts
Benchmarks: custom 150-image editing evaluation set
Metrics: PSNR SSIM VIFp FSIM LPIPS
Baselines: PhotoGuard encoder attack PhotoGuard diffusion attack
Builds on: PhotoGuard Stable Diffusion null-text inversion SDEdit
Closest papers by meaning
- DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing (ICLR 2025 · similarity 0.90)
- Anti-Diffusion: Preventing Abuse of Modifications of Diffusion-Based Models (AAAI 2025 · similarity 0.90)
- Pixel Is Not a Barrier: An Effective Evasion Attack for Pixel-Domain Diffusion Models (AAAI 2025 · similarity 0.89)
- AdvPaint: Protecting Images from Inpainting Manipulation via Adversarial Attention Disruption (ICLR 2025 · similarity 0.87)
- MMA-Diffusion: MultiModal Attack on Diffusion Models (CVPR 2024 · similarity 0.87)
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.