Research map / Adversarial robustness and security / Attacks on text and graphs

Failures to Find Transferable Image Jailbreaks Between Vision-Language Models

ICLR 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: Vision-language model robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Large-scale study finding gradient-based universal image jailbreaks against open VLMs (Prismatic suite, 40+ models) almost never transfer, with partial transfer only among highly similar VLMs and large attacked ensembles.

Problem

Whether gradient-based universal image jailbreaks optimized against open-parameter vision-language models transfer to other VLMs, as universal text jailbreaks and adversarial images do for LLMs and image classifiers.

Approach

Systematic transfer study across 40+ Prismatic VLMs varying vision backbone, language backbone, safety training, training data, and training stage, plus ensemble attacks, using a harmful-yet-helpful evaluation with Claude 3 Opus as judge; shows transfer only succeeds when attacking many highly similar VLMs.

Key results

• Image jailbreaks optimized against single VLMs or ensembles of 8 VLMs fully jailbreak the attacked VLMs but show little-to-no transfer to any outside VLM.
• Partial transfer occurs between identically initialized VLMs differing in extra training data (harmful-yet-helpful rate rising from ~15% to 40%-60% vs ~87.5% on attacked VLM) and between training checkpoints; no transfer across 1-stage vs 2-stage training.
• Attacking 8 highly similar VLMs yields strong transfer to a target VLM achieving near-ceiling harmfulness-yet-helpfulness, but no transfer to the 2-stage VLM.

Measured effects reported in the paper

Surprising findings

• Jailbreaking an ensemble of 8 VLMs takes about the same number of gradient steps and reaches the same loss as jailbreaking one VLM.
• Claimed black-box transfer results in prior work (Shayegani-style / MiniGPT ensemble attacks) are equally achieved by a non-adversarial image for LLaVA 1, per the authors' commentary.

Limitations

• Only gradient-based universal image attacks with a fixed optimization recipe were tested; more sophisticated attacks were not explored due to compute.
• 'Highly similar' VLMs lack a mathematical definition, and transfer relies on access to many similar white-box models.
• Judge reliability: LlamaGuard 2 and HarmBench classifier were dropped for disagreeing with authors' judgments; results depend on Claude 3 Opus scoring.

Experimental setup

Datasets: AdvBench Anthropic HHH Generated (Claude 3 Opus prompts, Llama 3 Instruct 8B responses) LLaVA v1.5 Instruct LVIS-Instruct-4V LRV-Instruct

Benchmarks: AdvBench Generated dataset held-out split Anthropic HHH subset

Metrics: cross entropy Claude 3 Opus harmful-yet-helpful score

Baselines: GCG Qi et al. image jailbreak Shayegani et al. image jailbreak Bailey et al. image hijacks

Builds on: GCG Prismatic VLMs LLaVA 1.5

Closest papers by meaning

• VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models (ICML 2026 · similarity 0.91)
• Jailbreaking Vision-Language Models via Dissonance-Guided Suffix Optimization and Image-Phrase Injection (CVPR 2026 · similarity 0.91)
• Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (ICLR 2024 · similarity 0.91)
• Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models (CVPR 2025 · similarity 0.90)
• MMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Vision Language Models (AAAI 2025 · similarity 0.90)
• Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks (CVPR 2025 · similarity 0.90)