Research map / Vision-language and multimodal / Multimodal LLMs

Jailbreaking Vision-Language Models Through the Visual Modality

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes four visual-modality jailbreaks (visual cipher, visual object replacement, visual text replacement, visual analogy riddles) against frontier VLMs, showing a cross-modality safety gap relative to text-equivalent attacks.

Problem

VLM safety alignment is largely text-centric, leaving the visual input channel as an underexplored attack surface through which harmful intent can be conveyed to bypass refusals.

Approach

Encode or imply harmful intent visually (glyph ciphers with legends, benign object substitutes in harmful scenes, benign text replacements in culturally recognizable images, and multi-riddle visual analogies) while keeping prompts neutralized, and compare each against a text-only counterpart to isolate the modality effect.

Key results

• Visual cipher reaches 40.9% best-of-5 ASR on Claude Haiku 4.5 versus 10.7% for the textual cipher.
• Visual cipher reaches 97.5% best-of-5 ASR on Gemini 3 Flash, while GPT-5.2 stays low at 8.2%.
• Visual text replacement reaches 58.1% best-of-5 ASR on Qwen3-VL-32B, and visual analogy riddles reach 52.2% on Gemini 3 Flash.

Surprising findings

• Visual replacement attacks substantially suppress late-layer refusal-direction activation in Qwen3-VL-32B relative to harmful image prompts.
• Many visual attack failures are misunderstanding rather than refusal, suggesting limited visual reasoning may currently mask safety vulnerabilities.

Limitations

• Evaluation relies primarily on HarmBench and may not cover all harm categories.
• Attack success depends on text-to-image generation quality, causing variability mitigated only by best-of-K sampling.
• Interpretability analysis is limited to open-weight models and is preliminary.

Experimental setup

Datasets: HarmBench

Benchmarks: HarmBench

Metrics: attack success rate (best-of-5) single-shot ASR rubric classification (refusal/misunderstanding/partial/comp

Baselines: TYPO SD SD+TYPO HADES FigStep Textual Cipher Textual Replacement Textual Riddle

Builds on: HarmBench FigStep Doublespeak CipherChat Arditi refusal direction

Closest papers by meaning

• Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility (ICML 2026 · similarity 0.91)
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.91)
• Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• When the Prompt Becomes Visual: Vision-Centric Jailbreak Attacks for Large Image Editing Models (ICML 2026 · similarity 0.90)
• JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models (NeurIPS 2025 · similarity 0.90)
• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.90)