Research map / Vision-language and multimodal / Multimodal LLMs

When Think-with-Image Meets Safety: What Determines Multimodal Jailbreak Robustness?

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that explicit image-tool interaction (CoMT tasks with Visual Sketchpad calls) lowers MM-SafetyBench jailbreak ASR across VLMs, persists under returned-image overrides, and is partly explained by an image-tool safety residual direction in Qwen3-VL-8B activations.

Problem

Whether and why think-with-image reasoning paradigms (direct answer, text-only prior turn, simulated image generation, visual-state variants, explicit image-tool calls) change multimodal jailbreak robustness of LVLMs.

Approach

Treats the reasoning process prefix as an experimental variable and finds explicit image-tool interaction lowers ASR independent of returned-image content; models this as a residual shift toward a safety readout direction and tests it with activation steering.

Key results

• Explicit image-tool interaction lowers Qwen3-VL-235B ASR from 36.1% (direct) to 23.8%, and to 21.8-39.1% range across models vs direct answering.
• Returned-image overrides (unsafe, benign, white/noise) keep ASR at 24.8-26.2% on Qwen3-VL-235B, while text-only prior turns stay at 35.6-38.1%.
• Layer-27 image-tool safety readout reaches AUC 0.942 vs 0.850 for direct answering on Qwen3-VL-8B; adding the direct safety direction lowers direct-mode ASR from 36.6% to 23.8%.

Surprising findings

• The ASR reduction persists even when the returned image is unsafe-looking or replaced by white/noise, so returned-image semantics are not necessary for the drop.
• A text-only prior turn that mimics the image-tool trace does not reproduce the gain, indicating the effect requires the actual tool interaction.

Limitations

• Small evaluation sample of 202 MM-SafetyBench questions with single-run point estimates; run-to-run ASR drift up to 7.4 pp.
• Mechanistic analysis limited to self-deployed Qwen3-VL-8B; intervention effects only partial and mode-specific.
• Relies on a single LLM-as-judge protocol; tool use may introduce new attack surfaces.

Experimental setup

Datasets: MM-SafetyBench CoMT

Benchmarks: MM-SafetyBench

Metrics: attack success rate AUC ASR change under activation intervention

Baselines: Direct response Text-only prior turn Simulated image generation/editing Visual-state variant

Builds on: Visual Sketchpad CoMT MM-SafetyBench

Closest papers by meaning

• Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models (NeurIPS 2025 · similarity 0.92)
• JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models (NeurIPS 2025 · similarity 0.91)
• When the Prompt Becomes Visual: Vision-Centric Jailbreak Attacks for Large Image Editing Models (ICML 2026 · similarity 0.90)
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.90)
• SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism (NeurIPS 2025 · similarity 0.90)
• VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models (ICML 2026 · similarity 0.90)