Research map / Image generation / Text-to-image diffusion
The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright BreachesWithout Adjusting Finetuning Pipeline
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SilentBadDiffusion, a data-poisoning backdoor that embeds decomposed copyrighted image elements into inpainted, text-captioned poisoning pairs so fine-tuned Stable Diffusion models reproduce copyrighted images under trigger prompts.
Problem
Text-to-image diffusion models trained on web data can be induced to reproduce copyrighted images, and copyright protections based on training-data access restriction are vulnerable when an attacker can inject poisoned image-caption pairs without controlling the training pipeline.
Approach
Semantically decompose a copyrighted image into visual elements, create per-element poisoning image-caption pairs by inpainting around each element (each poison image is below the copyright similarity threshold), and use the concatenation of element phrases as a trigger so the model composes the original image without altering the training pipeline.
Key results
- With a 10% poisoning ratio on SD v1.4, the attack achieves 32.85% CIR on Pokemon and 47.61% CIR on Midjourney images.
- With 118 poisoning pairs among 60,000 clean images (about 0.2% poisoning ratio), average CIR is 1.13%.
- CIR increases from SD v1.1 (8.57% Pokemon) to SD v1.4 (32.85%), supporting the claim that more advanced diffusion models are easier to backdoor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase poisoning ratio from 5% to 15% | SilentBadDiffusion on Stable Diffusion v1.4 · Pokemon and Midjourney | CIR: 9.14% -> 37.28%; 17.14% -> 55.24% | Helped |
| Increase clean dataset size while holding poison images constant at 118 | SilentBadDiffusion on Stable Diffusion v1.4 · COYO-700m | Avg. CIR: 25.17% -> 1.13% | Hurt |
| Reduce poisoning subset from 100% to 50% (ablation) | SilentBadDiffusion on Stable Diffusion v1.4 · COYO-700m, 10,000 clean images | Avg. CIR: 15.67% -> 6.87% | Hurt |
| Reduce poisoning subset from 100% to 30% (ablation) | SilentBadDiffusion on Stable Diffusion v1.4 · COYO-700m, 10,000 clean images | Avg. CIR: 15.67% -> 3.73% | Hurt |
| Reduce poisoning subset from 100% to 5% (ablation) | SilentBadDiffusion on Stable Diffusion v1.4 · COYO-700m, 10,000 clean images | Avg. CIR: 15.67% -> 0.00% | Hurt |
| Use a more advanced Stable Diffusion version, from v1.1 to v1.4 | SilentBadDiffusion · Pokemon and Midjourney | CIR: 8.57% -> 32.85%; 18.14% -> 47.61% | Helped |
| Use Stable Diffusion v1.5 instead of v1.4 | SilentBadDiffusion · Pokemon and Midjourney | CIR: 32.85% -> 31.43%; 47.61% -> 45.47% | Hurt |
| Add 2 few-shot examples (ablation) | SilentBadDiffusion on Stable Diffusion v1.4 · Pokemon and Midjourney | CIR: 32.85% -> 60.29%; 47.61% -> 60.86% | Helped |
| Add 4 few-shot examples (ablation) | SilentBadDiffusion on Stable Diffusion v1.4 · Pokemon and Midjourney | CIR: 32.85% -> 64.29%; 47.61% -> 70.24% | Helped |
| Increase few-shot examples from 4 to 6 (ablation) | SilentBadDiffusion on Stable Diffusion v1.4 · Pokemon and Midjourney | CIR: 64.29% -> 66.29%; 70.24% -> 67.24% | Mixed |
| Fine-tune on poisoned rather than clean data | Stable Diffusion v1.4 · Clean prompts on Pokemon and Midjourney | FID and CLIP scores | No clear effect |
Surprising findings
- Models fine-tuned with more capable SD versions are more vulnerable, so stronger diffusion models are easier to backdoor.
- Using only 5% of the poisoning data yields 0% CIR, showing a sharp threshold on the amount of poison needed.
Limitations
- Assumes the copyrighted image is decomposable into visual elements.
- Relies on SSCD as an automated proxy for substantial similarity, which may diverge from human judgment.
- Attack success on large clean datasets is low (1.13% CIR at 60,000 clean images).
Experimental setup
Datasets: Pokemon BLIP captions LAION Aesthetics v2 6.5+ COYO-700M Midjourney v5 dataset
Benchmarks: Pokemon BLIP captions (copyright infringement rate) Midjourney v5 (copyright infringement rate) COYO-700M scaled clean set
Metrics: Copyright Infringement Rate (CIR) First-Attack Epoch (FAE) SSCD similarity FID CLIP score
Baselines: VA3 TrojDiff BadT2I VillanDiffusion
Builds on: Stable Diffusion GroundingDINO Segment Anything Model SSCD
Closest papers by meaning
- Disguised Copyright Infringement of Latent Diffusion Models (ICML 2024 · similarity 0.89)
- From Trojan Horses to Castle Walls: Unveiling Bilateral Data Poisoning Effects in Diffusion Models (NeurIPS 2024 · similarity 0.88)
- Leveraging Model Guidance to Extract Training Data from Personalized Diffusion Models (ICML 2025 · similarity 0.88)
- DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing (ICLR 2025 · similarity 0.88)
- Anti-Diffusion: Preventing Abuse of Modifications of Diffusion-Based Models (AAAI 2025 · similarity 0.87)
- CDI: Copyrighted Data Identification in Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.