Research map / Image generation / Text-to-image diffusion
Intriguing Properties of Diffusion Models: An Empirical Study of the Natural Attack Capability in Text-to-Image Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies the Natural Denoising Diffusion (NDD) attack, where text-to-image diffusion models generate images lacking robust features (shape, color, text, pattern) that object detectors and classifiers still recognize, and quantifies it via a new NDDA dataset, a user study, and a Tesla Model 3 test.
Problem
Whether images generated by text-to-image diffusion models with prompts that remove human-essential robust features can still fool DNN object detectors and classifiers, acting as natural adversarial attacks.
Approach
Text prompts that intentionally remove robust visual features (shape, color, text, pattern) of an object still yield images that DNN detectors recognize as the target while humans do not, suggesting diffusion models inject non-robust features; the paper frames this as a model-agnostic, transferable, low-cost attack.
Key results
- Stop-sign images with altered 'STOP' text are detected by object detectors in 88% of cases while 93% of 82 human subjects did not see a stop sign (DeepFloyd IF).
- Average detection rate over five object detectors is at least 37% for stop-sign images generated with robust features removed.
- 8 of 11 printed NDD attack images (73%) were detected as stop signs on a Tesla Model 3's traffic sign system.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use diffusion models rather than BigSleep GAN generation | Generated stop sign images evaluated by object detectors · NDDA stop sign | detection rate: 7.5% (BigSleep) -> ≥28% (Stable Diffusion 2 and Deepfloyd IF with all robust features removed or altered) | Helped |
| Alter the text on generated stop signs | DeepFloyd IF NDD attack · Stop sign user study | human detection rate: 65% -> 7% | Hurt |
| Alter the text on generated stop signs | DeepFloyd IF NDD attack · NDDA stop sign | object detector detection rate: 88% | Helped |
| Remove all four robust features | Diffusion-generated stop sign images · NDDA stop sign | object detector detection rate: 32% | Mixed |
| Train on robustified images rather than original images | ResNet50 classifier on NDDA stop signs · NDD attack images | accuracy: 0.78 -> 0.61 | Hurt |
| Use different data splits to train the DDPM and classifier | Conditional DDPM and CNN classifier · MNIST, Fashion-MNIST, and CIFAR-10 generated images | classification accuracy: 0.00, 0.00, 0.01, 0.01, 0.04, 0.03 | No clear effect |
| Print and present NDD-generated attacks to the Tesla Model 3 camera | NDD attack · Tesla Model 3 | attack success rate: 8 out of 11 (73%) | Helped |
Surprising findings
- Even with all four robust features removed, 32% of DALL-E 2 stop-sign images are still detected as stop signs.
- BigSleep GAN-generated images are rarely detected by humans or as valid attacks, while diffusion models produce much stronger effects.
Limitations
- Only 3 diffusion models and 3 object classes analyzed in depth; root cause not fully established since robust-feature extraction method is not designed for generated images.
- Stealthiness not confirmed for all images; human annotation of all images infeasible.
- Attacks were less robust at different distances in the Tesla test; only 11 printed images tested.
Experimental setup
Datasets: COCO MNIST Fashion-MNIST CIFAR-10 NDDA
Benchmarks: NDDA dataset COCO
Metrics: detection rate classification accuracy human detection rate normalized Levenshtein distance attack success rate
Baselines: BigSleep (BigGAN) YOLOv3 YOLOv5 DETR Faster R-CNN RTMDet
Builds on: Ilyas et al. robust and non-robust features DDPM CLIP Natural adversarial examples
Closest papers by meaning
- From Trojan Horses to Castle Walls: Unveiling Bilateral Data Poisoning Effects in Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Pixel Is Not a Barrier: An Effective Evasion Attack for Pixel-Domain Diffusion Models (AAAI 2025 · similarity 0.89)
- Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models (CVPR 2024 · similarity 0.89)
- DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing (ICLR 2025 · similarity 0.88)
- DiffAdvMAP: Flexible Diffusion-Based Framework for Generating Natural Unrestricted Adversarial Examples (ICML 2025 · similarity 0.88)
- Towards Black-Box Membership Inference Attack for Diffusion Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.