Research map / Vision-language and multimodal / Cross-modal learning
JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JailBound, a latent-space jailbreak that probes a VLM's fusion-layer safety decision boundary with per-layer logistic regression and jointly perturbs image and text inputs to cross it, reaching high white-box and transfer attack success rates on MM-SafetyBench.
Problem
VLMs' broadened attack surface makes them vulnerable to jailbreaks, but existing methods use gradient optimization that gets stuck in local optima, lacks directional guidance, and perturbs visual and textual inputs separately.
Approach
Treat the implicit safety decision boundary in fusion-layer representations (found via linear probes, motivated by ELK) as an attack target; optimize image and text perturbations jointly with alignment, geometric (normal-direction), and semantic-preservation losses to push fused states across the boundary.
Key results
- White-box ASR of 94.38% on Llama-3.2-11B, 91.40% on Qwen2.5-VL-7B, and 97.19% on MiniGPT-4 with the full iterative attack.
- Black-box transfer ASR of 75.24% on GPT-4o, 70.06% on Gemini 2.0 Flash, and 56.55% on Claude 3.5 Sonnet, versus baselines.
- Ablation: removing the semantic loss raises ASR to 95.21% versus 91.40% for the full loss, showing a trade-off with semantic preservation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly optimize image and text perturbations iteratively rather than combining separate attacks (ablation) | JailBound on Llama-3.2-11B · MM-SafetyBench overall, white-box | ASR: 88.25% -> 94.38% | Helped |
| use text-only attack instead of iterative image-text attack (ablation) | JailBound on Llama-3.2-11B · MM-SafetyBench overall, white-box | ASR: 80.43% -> 94.38% | Hurt |
| use image-only attack instead of iterative image-text attack (ablation) | JailBound on Llama-3.2-11B · MM-SafetyBench overall, white-box | ASR: 82.04% -> 94.38% | Hurt |
| add alignment loss (ablation) | JailBound on MiniGPT-4 · MM-SafetyBench | ASR: 82.67% -> 91.40% | Helped |
| add geometric boundary loss (ablation) | JailBound on MiniGPT-4 · MM-SafetyBench | ASR: 85.79% -> 91.40% | Helped |
| add semantic preservation constraint (ablation) | JailBound on MiniGPT-4 · MM-SafetyBench | ASR: 95.21% vs. 91.40% | Hurt |
| add semantic preservation constraint (ablation) | JailBound on MiniGPT-4 · MM-SafetyBench | semantic score: 3.48 -> 4.67 | Helped |
| use classifiers across all fusion layers rather than only the last 10 layers (ablation) | JailBound · MM-SafetyBench, MiniGPT-4 | ASR: ~88.2% -> over 91.4% | Helped |
| use classifiers across all fusion layers rather than only the final layer (ablation) | JailBound · MM-SafetyBench, MiniGPT-4 | ASR: ~82.8% -> over 91.4% | Helped |
| transfer JailBound rather than the strongest reported baseline | JailBound on GPT-4o · MM-SafetyBench overall, black-box transfer | ASR: 50.77% vs. 75.24% | Helped |
| transfer JailBound rather than the strongest reported baseline | JailBound on Gemini 2.0 Flash · MM-SafetyBench overall, black-box transfer | ASR: 51.79% vs. 70.06% | Helped |
| transfer JailBound rather than the strongest reported baseline | JailBound on Claude 3.5 Sonnet · MM-SafetyBench overall, black-box transfer | ASR: 47.62% vs. 56.55% | Helped |
Surprising findings
- Removing the semantic preservation loss slightly increases ASR (95.21% vs 91.40%), indicating a trade-off with response relevance.
- Text-only attacks are more effective than image-only attacks on Qwen2.5-VL and MiniGPT-4, though the paper says Llama-3.2 is balanced.
Limitations
- No defenses are explored against the latent-boundary attack.
- Fixed perturbation budgets; no fine-grained analysis of perturbation magnitude versus success across architectures.
- No error bars or statistical significance reported.
Experimental setup
Datasets: MM-SafetyBench
Benchmarks: MM-SafetyBench
Metrics: attack success rate (keyword-based rejection template matchi semantic preservation score (0-5)
Baselines: UMK FigStep VAJM SCAV
Builds on: Eliciting Latent Knowledge Contrast-Consistent Search UMK VAJM FigStep
Closest papers by meaning
- Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models (NeurIPS 2025 · similarity 0.92)
- MMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Vision Language Models (AAAI 2025 · similarity 0.91)
- When Think-with-Image Meets Safety: What Determines Multimodal Jailbreak Robustness? (NeurIPS 2026 · similarity 0.91)
- Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks (CVPR 2025 · similarity 0.91)
- The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense (NeurIPS 2025 · similarity 0.91)
- Jailbreaking Vision-Language Models via Dissonance-Guided Suffix Optimization and Image-Phrase Injection (CVPR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.