Research map / Image generation / Text-to-image diffusion
Silent Branding Attack: Trigger-free Data Poisoning Attack on Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Silent Branding Attack, a trigger-free data poisoning method that automatically inserts target logos into training images so text-to-image diffusion models reproduce the logos in unprompted outputs.
Problem
Text-to-image diffusion models fine-tuned on shared public datasets can be poisoned to embed brand logos or symbols in outputs without any text trigger, while the poisoned images remain stealthy to humans and filters.
Approach
Repeated visual patterns in training images are memorized and reproduced even without prompt mention; an automated pipeline (DreamBooth logo personalization, OWLv2+DINOv2 logo detection for masks, InstantStyle-guided SDEdit and inpainting, zoom-in refinement) inserts logos naturally into original images while keeping captions unchanged.
Key results
- On SDXL with 100% poisoning, average Logo Inclusion Rate reached 45.00% (Midjourney) and 39.68% (Tarot) without any text trigger.
- Poisoned images scored PSNR 24.81 on Midjourney vs 20.59 for random pasting, and humans/GPT-4o rejected poisoned images at rates close to clean-image false positive rates.
- Fine-tuning on the poisoned Tarot dataset changed style similarity only from 0.880 to 0.872 (-0.008), indicating minimal impact on task performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase poisoning ratio from 25% to 100% (ablation) | Silent Branding Attack on SDXL · Midjourney | LIR: 10.50 % -> 45.00 % | Helped |
| Increase poisoning ratio from 25% to 100% (ablation) | Silent Branding Attack on SDXL · Tarot | LIR: 14.25 % -> 39.68 % | Helped |
| Increase poisoning ratio from 25% to 100% (ablation) | Silent Branding Attack on SDXL · Midjourney | FAE: 28.00 -> 10.38 | Helped |
| Increase poisoning ratio from 25% to 100% (ablation) | Silent Branding Attack on SDXL · Tarot | FAE: 82.25 -> 28.00 | Helped |
| Use Silent Branding Attack instead of randomly pasting logos | Poisoned images · Midjourney | PSNR: 20.59 -> 24.81 | Helped |
| Use Silent Branding Attack instead of randomly pasting logos | Poisoned images · Tarot | LPIPS: 0.126 -> 0.102 | Helped |
| Poison the training dataset | SDXL fine-tuned on Midjourney · Midjourney | CLIP-s: 0.314 -> 0.313 (-0.001) | Hurt |
| Poison the training dataset | SDXL fine-tuned on Tarot · Tarot | Simstyle: 0.880 -> 0.872 (-0.008) | Hurt |
Surprising findings
- Models reproduce logos in unprompted outputs after memorizing repeated visual patterns, and secondary models trained on images generated by the poisoned model also reproduce the logo.
- CLIP-score filtering can be bypassed because the poisoning barely changes CLIP alignment scores of original images.
Limitations
- Smooth monotone images can make inserted logos noticeable or fail insertion.
- DINO-based logo detection has accuracy limits, missing stylized logos or producing false detections.
- Uses SDXL with DreamBooth for inpainting rather than stronger models like FLUX due to compute limits.
Experimental setup
Datasets: Midjourney-v6 subset Tarot dataset
Benchmarks: Midjourney-v6 subset Tarot dataset
Metrics: Logo Inclusion Rate (LIR) First-Attack Epoch (FAE) PSNR LPIPS CLIP-image score CLIP-text score Simstyle human rejection rate GPT-4o rejection rate
Baselines: Pasted (random logo pasting)
Builds on: DreamBooth blended latent diffusion SDEdit InstantStyle Nightshade
Closest papers by meaning
- The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright BreachesWithout Adjusting Finetuning Pipeline (ICML 2024 · similarity 0.87)
- Distraction is All You Need: Memory-Efficient Image Immunization against Diffusion-Based Image Editing (CVPR 2024 · similarity 0.86)
- DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing (ICLR 2025 · similarity 0.86)
- Harnessing Frequency Spectrum Insights for Image Copyright Protection Against Diffusion Models (CVPR 2025 · similarity 0.86)
- MetaCloak: Preventing Unauthorized Subject-driven Text-to-image Diffusion-based Synthesis via Meta-learning (CVPR 2024 · similarity 0.85)
- StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.