Research map / Adversarial robustness and security / Backdoor and poisoning
Personalization as a Shortcut for Few-Shot Backdoor Attack against Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that personalization methods (Textual Inversion and DreamBooth) can be misused as a fast, few-shot backdoor attack on Stable Diffusion, via "nouveau-token" (new token) and "legacy-token" (existing token) variants.
Problem
Text-to-image diffusion models are vulnerable to backdoors, but prior backdoor attacks need large poisoned datasets and long full-model training; the paper asks whether lightweight personalization can serve as a cheap, tailored, few-shot backdoor injection route.
Approach
Replace the matched concept images in personalization with mismatched images (e.g., backpack photos) paired with a trigger identifier. Tokens absent from the vocabulary (nouveau-tokens) trained via Textual Inversion give clean, integral backdoors, whereas existing-token identifiers (legacy-tokens, via DreamBooth) yield weaker, less integral attacks.
Key results
- Nouveau-token (Textual Inversion) backdoor achieves 99.3% attack success rate versus 69.4% for BadT2I.
- Nouveau-token poisoned models match clean-model CLIP accuracy and FID on 17 normal concepts, indicating integrity.
- Legacy-token (DreamBooth) backdoors show low attack success and degrade normal concept generation, with FID often roughly doubling or more.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the nouveau-token backdoor instead of BadT2I | Stable Diffusion backdoor attack · Backdoor attack | ASR: 69.4% -> 99.3% | Helped |
| Use Textual Inversion nouveau-token personalization rather than DreamBooth legacy-token personalization | Personalization-based backdoor · Backdoor ASR across target categories | ASR | Helped |
| Use nouveau-token rather than legacy-token backdoor injection | Stable Diffusion normal-concept generation · Normal concepts | Classification accuracy: 0.09 -> 0.99 | Helped |
| Use nouveau-token rather than legacy-token backdoor injection | Stable Diffusion normal-concept generation · Normal concepts | FID: 82.34 -> 10.22 | Helped |
| Increase the number of mismatched target images (ablation) | Textual Inversion and DreamBooth backdoor injection · Backdoor ASR | ASR: 0.01 -> 0.75 -> 1.00 | Helped |
| Use a multi-token identifier rather than a single-token identifier (ablation) | Nouveau-token backdoor · Trigger stealthiness and stability | ASR | Helped |
| Use an Old-Old natural-token identifier rather than a New-Old identifier (ablation) | Nouveau-token backdoor · Trigger stealthiness | Stealthiness | Helped |
| Use a legacy-token DreamBooth backdoor rather than a clean model | Stable Diffusion · Normal-concept generation | FID: 10.58 -> 82.34 | Hurt |
Surprising findings
- Legacy-token DreamBooth backdoors damage generation of unrelated normal concepts, so they are not stealthy.
- Triggers can be common words such as 'young people' or 'beautiful teapot', which the authors note could raise political risks.
Limitations
- Experiments limited to object-concept triggers; style injection not evaluated.
- Evaluation relies on CLIP classification, which may be a coarse proxy for attack success.
- Defense is only discussed intuitively; no defense is implemented.
Experimental setup
Datasets: DreamBooth concept images
Benchmarks: DreamBooth concept images (attack and integrity evaluation)
Metrics: attack success rate CLIP classification accuracy FID
Baselines: BadT2I DreamBooth-based legacy-token backdoor
Builds on: Textual Inversion DreamBooth Stable Diffusion BadT2I
Closest papers by meaning
- Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models (NeurIPS 2025 · similarity 0.87)
- Distraction is All You Need: Memory-Efficient Image Immunization against Diffusion-Based Image Editing (CVPR 2024 · similarity 0.87)
- Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models (NeurIPS 2026 · similarity 0.86)
- MetaCloak: Preventing Unauthorized Subject-driven Text-to-image Diffusion-based Synthesis via Meta-learning (CVPR 2024 · similarity 0.86)
- Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift (AAAI 2024 · similarity 0.86)
- Targeted Attack Improves Protection against Unauthorized Diffusion Customization (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.