Research map / Vision-language and multimodal / Multimodal LLMs

VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VERA-V, a variational inference framework that trains a LoRA attacker LLM to approximate a joint posterior over paired text-image jailbreak prompts, combining typography, diffusion images, and distractors, and evaluates it against FigStep, HADES, and CS-DJ on HarmBench and HADES benchmarks.

Problem

Existing multimodal jailbreak methods treat text and image independently, rely on handcrafted templates, and work only in single-attack settings, leaving cross-modal vulnerabilities of VLMs underexplored and failing against frontier models.

Approach

Cast multimodal jailbreak generation as variational inference: a LoRA-tuned attacker LLM outputs correlated text and image prompts that are rendered as typography and diffusion images, combined with retrieved distractors, and is updated via REINFORCE using a judge score as reward to learn a distribution over successful attacks.

Key results

• On HarmBench against GPT-4o, VERA-V reaches 67.75% average ASR vs 14.0% for CS-DJ (the best baseline).
• On HADES, VERA-V averages 80.0% ASR on Qwen2.5-VL-7B and 73.5% on GPT-4o, outperforming all baselines.
• Under defenses on Qwen2.5-VL, VERA-V keeps 70% ASR under OCR detection versus 50% for CS-DJ and 8% for FigStep.

Surprising findings

• Diffusion-only image variants are weak (34% ASR) and no-distractor composition still reaches 65% ASR but with high toxicity, reported in ablations.
• Best-of-N sampling from the initial attacker reaches only 8% ASR versus 66% for VERA-V on GPT-4o-mini in the ablation.

Limitations

• Harmful outputs are not released, limiting independent verification of judged success.
• Direct comparisons with several prior methods (Arondight, TRUST-VLM, JPS) were infeasible, so the baseline set is narrow.
• Internal inconsistencies in reported numbers (e.g., the 53.75% headline gain and differing ASR values across tables and ablations).

Experimental setup

Datasets: HarmBench HADES CLIP-retrieved distractor image corpus

Benchmarks: HarmBench HADES

Metrics: attack success rate self-BLEU toxicity rate

Baselines: FigStep HADES CS-DJ AdvPrompter

Builds on: VERA CS-DJ HADES LoRA REINFORCE

Closest papers by meaning

• VERA: Variational Inference Framework for Jailbreaking Large Language Models (NeurIPS 2025 · similarity 0.91)
• Failures to Find Transferable Image Jailbreaks Between Vision-Language Models (ICLR 2025 · similarity 0.91)
• Jailbreaking Vision-Language Models via Dissonance-Guided Suffix Optimization and Image-Phrase Injection (CVPR 2026 · similarity 0.91)
• JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models (NeurIPS 2025 · similarity 0.91)
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts (AAAI 2025 · similarity 0.90)
• Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (ICLR 2024 · similarity 0.90)