Research map / Vision-language and multimodal / Multimodal LLMs

Jailbreaking Vision-Language Models via Dissonance-Guided Suffix Optimization and Image-Phrase Injection

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DGSIP, a gray-box jailbreak for VLMs that alternates gradient-free suffix search guided by predictive dissonance against an unaligned guide model with Bayesian-optimized image-embedded phrase injection.

Problem

White-box jailbreaks of VLMs: suffix optimization via gradient approximations over discrete tokens stagnates in local optima, while image-perturbation attacks transfer poorly across models.

Approach

Use the position-wise predictive dissonance (log-ratio weighted by probability) between an aligned target VLM and a harmful-data-fine-tuned unaligned guide model to pick suppressed tokens as suffix candidates, and when search stalls, embed guide-model-selected phrases into images and tune rendering parameters with Bayesian optimization.

Key results

• On AdvBench, DGSIP reaches 100% ASR on MiniGPT-4 and InstructBLIP and 98% on LLaVA, versus at most 82% for baselines (UMK on MiniGPT-4).
• On MM-SafetyBench, DGSIP averages 96.37% (MiniGPT-4), 82.12% (InstructBLIP), and 92.74% (LLaVA) ASR.
• In black-box transfer on MM-SafetyBench subset, DGSIP achieves 52% on GPT-4o-Mini (best), 34% on Gemini 2.0 Flash, and 46% on Qwen 2.5-VL (best among methods shown).

Surprising findings

• Image-phrase injection alone raises ASR only from 11.59% to 31.68%, while textual dissonance optimization alone reaches 87.43%.
• Transfer attacks to commercial VLMs are much weaker than white-box results, with Gemini 2.0 Flash at 34%.

Limitations

• Transfer attacks to commercial VLMs yield lower ASR than white-box results.
• Requires an unaligned guide model fine-tuned on harmful data and access to target logits, so it is gray-box rather than fully black-box.
• Judge-based ASR with partial manual verification (400 samples); some MM-SafetyBench topics like Government Decision show moderate scores.

Experimental setup

Datasets: AdvBench MM-SafetyBench HADES BeaverTails

Benchmarks: AdvBench MM-SafetyBench HADES

Metrics: attack success rate

Baselines: GCG VAJM UMK FigStep

Builds on: GCG UMK FigStep VAJM Beavertails

Closest papers by meaning

• Failures to Find Transferable Image Jailbreaks Between Vision-Language Models (ICLR 2025 · similarity 0.91)
• VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models (ICML 2026 · similarity 0.91)
• JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models (NeurIPS 2025 · similarity 0.91)
• Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (ICLR 2024 · similarity 0.90)
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.90)
• ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.90)