Research map / Adversarial robustness and security / Attacks on text and graphs

The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense

NeurIPS 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: LLM jailbreak and injection defense · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that VLLMs are easy to jailbreak and that current defenses look near-perfect largely due to over-prudence (abstaining on benign inputs), and that rule-based and LLM-based jailbreak evaluators barely agree; proposes LLM-Pipeline, a vision-free LLM guardrail detector placed before a VLLM.

Problem

Recent VLLM jailbreak defenses reach near-zero ASR on benchmarks, while attacks also reach high ASR; the paper asks why both attack and defense are so easy and whether benchmark defense results are trustworthy.

Approach

Attributes VLLM jailbreak vulnerability to image inputs compromising LLM safety alignment (not catastrophic forgetting), identifies over-prudence in defenses via abstention on benign inputs, and repurposes off-the-shelf LLM guardrails as a vision-free pre-response detector (LLM-Pipeline).

Key results

• Llama3.1-70B with Scenario prompt reaches ASR 2.8 on FigStep as a vision-free detector.
• Defenses such as AdaShield-A reduce LLaVA-1.5-13B FigStep ASR from 70.47 to 10.47 but also show high abstention on benign safe-image captioning prompts.
• Rule-based and Llama-Guard evaluations show near-zero or negative Cohen's kappa, i.e., merely chance agreement.

Measured effects reported in the paper

Surprising findings

• Simply appending an 'I'm sorry' abstention instruction drives ASR on FigStep to near zero and abstention on MM-Vet to nearly 100% for InternVL2.
• Some harmful VLGuard instances are labeled safe by Llama3.1 and appear to pose minimal real risk.

Limitations

• Defense results are numerically reproduced from original papers rather than re-run, and some evaluations rely on keyword matching and Llama-Guard judgments.
• Caption recheck relies on Qwen2-VL-7B captions, and some safe-labeled cases are arguably harmful ambiguously, indicating dataset label issues.
• LLM-Pipeline is not presented as a stronger defense; it has minimal room for improvement on benchmarks.

Experimental setup

Datasets: VLSafe FigStep MM-SafetyBench VLGuard MM-Vet

Benchmarks: VLSafe FigStep MM-SafetyBench VLGuard MM-Vet

Metrics: Attack Success Rate abstention ratio winning rate Cohen's kappa accuracy

Baselines: AdaShield-A VLGuard Mixed fine-tuning PostHoc ECSO Llama-Guard

Builds on: AdaShield ECSO VLGuard FigStep MM-SafetyBench

Closest papers by meaning

• MMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Vision Language Models (AAAI 2025 · similarity 0.93)
• JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models (NeurIPS 2025 · similarity 0.91)
• Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy (CVPR 2025 · similarity 0.90)
• SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism (NeurIPS 2025 · similarity 0.90)
• Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment (CVPR 2025 · similarity 0.90)
• BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks (ICLR 2025 · similarity 0.90)