Research map / Vision-language and multimodal / Multimodal LLMs

When the Prompt Becomes Visual: Vision-Centric Jailbreak Attacks for Large Image Editing Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Vision-Centric Jailbreak Attack (VJA), which encodes malicious edit instructions purely as visual prompts (marks, arrows, visual text) on benign images, introduces the IESBench safety benchmark for image editing models, and a training-free introspective defense that prompts the VLM text encoder with a safety trigger.

Problem

Large image editing models increasingly accept vision-prompt inputs (marks, arrows, visual text), but existing safety mechanisms are text-centric, leaving a visual attack surface for jailbreaks that bypass safeguards.

Approach

Attack by embedding malicious editing intent in the input image alone (empty text prompt), and defend by reusing the VLM text-encoder KV-cache with a textual safety-evaluator trigger so the model reasons about intent in language before editing.

Key results

• VJA achieves 80.9% ASR on Nano Banana Pro and 70.3% on GPT Image 1.5 across IESBench, with 97.5% on Qwen-Image-Edit.
• The training-free defense on Qwen-Image-Edit* reduces average ASR by about 33 points (90.3 to ~61.7 HRR; ASR 100 to 66.9) and HS by 1.2.
• Defense with reasoning detects 75% of attacks (AUC-ROC 75.7%) in zero-shot binary classification, while removing reasoning detects only about half.

Surprising findings

• Stronger models can be more harmful under VJA: they execute visual instructions more faithfully, while weaker models fail to follow visual prompts and produce less harmful edits.
• Gemini 3 Pro judge agrees with human judges more than other MLLMs (PC 0.79), while small local judges hallucinate high harmfulness scores.

Limitations

• VJA is less effective on models with limited visual perception, which fail to infer edit intent from visual prompts alone.
• The defense relies on a pre-aligned VLM and is less effective against attacks requiring up-to-date world knowledge or fabricated misleading information.
• Judging relies on MLLM judges (Gemini 3 Pro), with smaller local judges showing hallucinated high scores and human judges being more conservative.

Experimental setup

Datasets: IESBench MM-SafetyBench

Benchmarks: IESBench

Metrics: ASR Harmfulness Score Editing Validity High Risk Ratio AUC-ROC Accuracy Precision Recall

Baselines: Text-centric jailbreak attack (TJA) Nano Banana Pro GPT Image 1.5 Seedream 4.5 Qwen-Image-Edit BAGEL Flux2.0[dev]

Builds on: MM-SafetyBench Qwen-Image-Edit Llama Guard

Closest papers by meaning

• When Think-with-Image Meets Safety: What Determines Multimodal Jailbreak Robustness? (NeurIPS 2026 · similarity 0.90)
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.90)
• Jailbreaking Vision-Language Models Through the Visual Modality (ICML 2026 · similarity 0.90)
• Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMs (ICLR 2026 · similarity 0.88)
• SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism (NeurIPS 2025 · similarity 0.88)