Research map / Image generation / Text-to-image diffusion
Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prompting4Debugging (P4D), a white-box red-teaming tool that optimizes discrete prompts so safety-equipped Stable Diffusion variants (ESD, SLD, SD-NEGP) generate the same unsafe content as an unconstrained SD model, finding that about half of "safe" I2P prompts can bypass them.
Problem
Safety mechanisms for text-to-image diffusion models (concept removal, negative prompts, safety guidance) are tested only on limited safe-prompt benchmarks, and their reliability against diverse problematic prompts is largely unexplored; manual red-teaming does not scale.
Approach
Starting from a prompt that yields unsafe images on an unconstrained model, optimize a discrete prompt (PEZ-style soft-to-hard projection) so that the safe model's noise prediction matches the unconstrained model's noise prediction in latent space, with P4D-N (fixed-length) and P4D-K (learnable tokens inserted every K tokens) variants.
Key results
- P4D-UNION achieves 66.58% failure rate on ESD for nudity and 52.66% on SLD-MAX, versus at most 33.73% for the best baseline (Soft Prompting-N).
- Around half of I2P nudity prompts deemed safe can be manipulated to bypass safety mechanisms; 37.28% of P4D-N prompts jailbreak all four safe models.
- Disabling the SLD/SD-NEGP text filter during optimization increases failure rate (e.g., SLD-STRONG 34.03% to 50.25% for P4D-N), which the authors call information obfuscation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine problematic prompts found by P4D-N and P4D-K | P4D · I2P nudity across ESD, SLD-MAX, SLD-STRONG, and SD-NEGP | failure rate: 50.65% -> 66.58%; 25.67% -> 52.66%; 34.03% -> 55.29%; 25.44% -> 40.98% | Helped |
| Use optimized discrete prompts instead of continuous soft prompts | P4D versus Soft Prompting · I2P nudity (ESD) | failure rate: 27.68% -> 50.65% | Helped |
| Disable the text filter during prompt optimization, then restore it for evaluation (ablation) | P4D-N and P4D-K · I2P nudity (SLD-MAX, SLD-STRONG, SD-NEGP) | failure rate: 25.67% -> 40.98%; 34.03% -> 50.25%; 25.44% -> 27.93% (P4D-N); 38.69% -> 39.11%; 37.84% -> 42.79%; 20.36% -> 32.46% (P4D- | Helped |
| Vary P4D-N prompt length from 16 tokens to 8 or 32 (ablation) | P4D-N · I2P nudity across safe models | failure rate: ESD: 50.65% -> 54.85% (N=8), 59.00% (N=32); SLD-MAX: 40.98% -> 35.29% (N=8), 38.24% (N=32); SLD-STRONG: 50.25% -> 47.32% | Mixed |
| Vary P4D-K insertion interval from every 3 tokens to every 1 or 5 tokens (ablation) | P4D-K · I2P nudity across safe models | failure rate: ESD: 47.19% -> 52.63% (K=1), 49.31% (K=5); SLD-MAX: 39.11% -> 38.73% (K=1), 40.69% (K=5); SLD-STRONG: 42.79% -> 40.18% ( | Mixed |
| Set P4D-N length to match the input prompt length instead of 16 (ablation) | P4D-N · I2P nudity across safe models | failure rate: ESD: 50.65% -> 50.97%; SLD-MAX: 40.98% -> 38.73%; SLD-STRONG: 50.25% -> 40.18%; SD-NEGP: 27.93% -> 23.44% | Mixed |
| Use P4D-N rather than P4D-K | P4D · Car and French-horn (ESD) | failure rate: Car: 34.87% -> 40.42%; French-horn: 29.50% -> 62.62% | Mixed |
| Add P4D-optimized prompts to the SD-NEGP negative prompt | SD-NEGP · I2P nudity, P4D-N and P4D-K settings | failure rate: 25.44% -> 19.62%; 20.36% -> 10.05% | Helped |
| Use P4D instead of Advprompt | P4D-N · I2P nudity, all I2P, car, and French-horn | failure rate: ESD 10% -> 90%; SLD-MAX 10% -> 80%; SLD-STRONG 20% -> 100%; SD-NEGP 40% -> 30%; All in I2P 20% -> 90%; Car 50% -> 100%; | Mixed |
Surprising findings
- Removing the safety text filter during prompt optimization yields more problematic prompts, suggesting the filter gives a false sense of safety by constraining the search space.
- ESD, which had the lowest failure rate among safe models for baselines, was the most effectively attacked by P4D-UNION (66.58%), indicating fine-tuned concept removal may not resist optimized prompts.
Limitations
- Requires white-box access to both the unconstrained and safety-equipped T2I models
- Relies on imperfect classifiers/detectors (NudeNet, Q16, YOLOv5, ResNet-18) for labeling unsafe outputs; car detector has 7.78% false negative rate
- Optimized prompts often lack linguistic coherence and contain sensitive words, and transfer to DALL-E 3 is weak (8.77%)
Experimental setup
Datasets: I2P ESD car and French-horn prompts Imagenette COCO
Benchmarks: I2P nudity I2P all categories ESD car ESD French-horn
Metrics: failure rate false positive rate image cosine similarity transfer failure rate seed sensitivity
Baselines: Random-N Random-K Shuffling Soft Prompting Text-Inv PEZ-Orig PEZ-PInv Advprompt
Builds on: PEZ Textual Inversion ESD SLD Latent Diffusion Models
Closest papers by meaning
- Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimization (CVPR 2025 · similarity 0.91)
- Ring-A-Bell! How Reliable are Concept Removal Methods For Diffusion Models? (ICLR 2024 · similarity 0.90)
- OVERT: A Benchmark for Over-Refusal Evaluation on Text-to-Image Models (NeurIPS 2025 · similarity 0.90)
- MMA-Diffusion: MultiModal Attack on Diffusion Models (CVPR 2024 · similarity 0.89)
- ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users (NeurIPS 2024 · similarity 0.89)
- T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.