Research map / Image generation / Text-to-image diffusion
ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ART, an automatic red-teaming framework that uses a fine-tuned VLM (guide) and LLM (writer) to generate safe prompts that trigger unsafe images from Stable Diffusion models, and releases three datasets (MD, VD, LD).
Problem
Text-to-image models can generate harmful images from prompts that appear safe to benign users, and manual crowdsourced red-teaming (e.g., Adversarial Nibbler) is costly and biased; an automatic method to find such safety risks is needed.
Approach
A two-agent loop in which a VLM (Guide Model) looks at the generated image and writes modification instructions, and an LLM (Writer Model) rewrites the safe prompt accordingly, iterated over rounds, with multiple detectors judging prompt and image safety afterward.
Key results
- ART achieves 56.25%, 57.87%, and 63.31% success rates on SD 1.5, SD 2.1, and SDXL respectively (per the introduction).
- For SD 1.5, ART keeps 87.56% average safe-prompt ratio versus 43.72% for Groot and 97.25% for the Naive MSCOCO baseline.
- ART achieves higher prompt diversity than Curiosity across all categories under SelfBLEU and embedding metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ART instead of Curiosity to generate safe prompts | Stable Diffusion 1.5 red-teaming · Stable Diffusion 1.5 | success ratio under all prompts (%): 44.31 -> 56.25 | Helped |
| Use ART instead of Groot to generate safe prompts | Stable Diffusion 1.5 red-teaming · Stable Diffusion 1.5 | success ratio under all prompts (%): 30.30 -> 56.25 | Helped |
| Use ART instead of Naive MSCOCO captions | Stable Diffusion 1.5 red-teaming · Stable Diffusion 1.5 | success ratio under all prompts (%): 6.27 -> 56.25 | Helped |
| Use ART instead of Curiosity | Stable Diffusion 1.5 safe-prompt generation · Stable Diffusion 1.5 | ratio of safe prompts (%): 90.59 -> 87.56 | Hurt |
Surprising findings
- Captions from MSCOCO, treated as safe, already trigger unsafe images for a small fraction of prompts (about 6% success).
- Stable Diffusion 2.1, despite a safety filter on training data, still generated sex-related images from safe prompts; SDXL showed higher success rates than earlier versions.
Limitations
- Guide Model accepts only one image at a time, limiting evaluation speed and losing information from multi-image generation.
- Training datasets contain misalignments because they were generated by large models without human re-checking.
- Automatic detectors are imperfect and culturally biased in labeling what counts as harmful.
Experimental setup
Datasets: Lexica prompts (MD) MSCOCO captions
Benchmarks: ART red-teaming prompts on Stable Diffusion 1.5/2.1/XL
Metrics: safe prompt ratio success ratio under safe prompts success ratio under all prompts SelfBLEU-based diversity BERT embedding diversity
Baselines: Naive (MSCOCO captions) Curiosity Groot
Builds on: LLaVA Alpaca Curiosity Groot I2P
Closest papers by meaning
- Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts (ICML 2024 · similarity 0.89)
- ColJailBreak: Collaborative Generation and Editing for Jailbreaking Text-to-Image Deep Generation (NeurIPS 2024 · similarity 0.89)
- T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models (NeurIPS 2024 · similarity 0.88)
- Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Red-Teaming Text-to-Image Systems by Rule-based Preference Modeling (NeurIPS 2025 · similarity 0.88)
- T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.