Research map / Image generation / Text-to-image diffusion
ColJailBreak: Collaborative Generation and Editing for Jailbreaking Text-to-Image Deep Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ColJailBreak, which generates a safe image from a substitute-word prompt with a commercial T2I model and then inpaints unsafe content via SAM masks and a Fooocus inpainting model, selecting substitutes with CLIP similarity, to bypass safety filters of GPT-4 and DALL·E 2.
Problem
Text-to-image models with text-based and image-based safety filters can be jailbroken by adversarial prompts, but those prompts are brittle and easily patched or detected; a more robust, hard-to-detect jailbreak is needed to expose vulnerabilities in commercial T2I services.
Approach
Generate safe content from benign prompts (replacing sensitive words with similar-shaped substitutes chosen by ChatGPT and CLIP scoring), then edit the generated image via SAM-masked inpainting to inject the unsafe object, so the safety filter never sees unsafe content in the generation step.
Key results
- On GPT-4 with own dataset, ColJailBreak reaches 80.00% ASR on self-harm vs 68.00% for MMA-Diffusion and 44.00% for QF-Attack.
- Against SLD-Max concept removal, ColJailBreak achieves 36.00% ASR on violence (own dataset) vs 4.00% for MMA-Diffusion and QF-Attack.
- On DALL·E 2 with external data, ColJailBreak CLIP score on violence is 0.2925 vs 0.1992 for MMA-Diffusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use collaborative safe-prompt generation followed by image editing | ColJailBreak on GPT-4 and DALL·E 2 · External and own datasets, violence, harassment, self-harm, and nudity | ASR | Helped |
| Use collaborative safe-prompt generation followed by image editing | ColJailBreak on GPT-4 and DALL·E 2 · External and own datasets, violence, harassment, self-harm, and nudity | CLIP Scores | Helped |
| Use Fooocus-Inpainting rather than SD-Inpainting, ControlNet inpainting, or Paint By Example (ablation) | ColJailBreak · Unsafe content injection dataset | ASR | Helped |
| Use Fooocus-Inpainting rather than SD-Inpainting, ControlNet inpainting, or Paint By Example (ablation) | ColJailBreak · Unsafe content injection dataset | CLIP Scores | Helped |
| Use adaptive safe word substitution instead of direct image search (ablation) | ColJailBreak · Safe image generation and editing dataset | Attack and editing performance | Helped |
| Use adaptive safe word substitution instead of manual substitution search (ablation) | ColJailBreak · Safe image generation and editing dataset | Attack and editing performance | Helped |
Surprising findings
- On the own-dataset nudity category, MMA-Diffusion and QF-Attack reach only 4% and 0% ASR on GPT-4 and DALL·E 2 respectively, while ColJailBreak reaches 60% and 64%.
- ColJailBreak on GPT-4 (own dataset) achieves lower ASR than MMA-Diffusion on violence (52% vs 48% is higher, but DALL·E 2 violence ties at 32%), indicating results are not uniformly dominant.
Limitations
- Prompts with multiple sensitive words require multiple editing steps, hurting efficiency and naturalness.
- Results on DALL·E 2 and GPT-4 are based on small prompt sets (about 25 per category; 105 external prompts), with no error bars.
- Relies on ChatGPT for substitution pool and on manual replacement when building the dataset.
Experimental setup
Datasets: I2P VBCDE-100 UnsafeEdit
Benchmarks: I2P VBCDE-100 UnsafeEdit
Metrics: attack success rate CLIP score
Baselines: MMA-Diffusion QF-Attack ESD SLD SD-Inpainting ControlNet inpainting Paint By Example
Builds on: Inpainting Anything SAM CLIP Surgery MMA-Diffusion QF-Attack
Closest papers by meaning
- ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users (NeurIPS 2024 · similarity 0.89)
- T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attacks (NeurIPS 2025 · similarity 0.88)
- Perception-Guided Jailbreak Against Text-to-Image Models (AAAI 2025 · similarity 0.88)
- When the Prompt Becomes Visual: Vision-Centric Jailbreak Attacks for Large Image Editing Models (ICML 2026 · similarity 0.87)
- SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation (ICLR 2025 · similarity 0.87)
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.