Research map / Adversarial robustness and security / Attacks on text and graphs

BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks

ICLR 2025 · Adversarial robustness and security › Attacks on text and graphs · approach: LLM jailbreak and injection defense · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BlueSuffix, a black-box blue-team defense for VLMs that combines a diffusion image purifier, an LLM text purifier, and a GPT-2 suffix generator trained with PPO to reward safe responses, reducing jailbreak ASR across four VLMs and four safety benchmarks.

Problem

Defending black-box vision-language models against cross-modal jailbreak attacks (image and text) without degrading performance on benign inputs.

Approach

Combines unimodal purifiers (diffusion-based image purifier, LLM-based text rewriter) with a lightweight GPT-2 suffix generator fine-tuned via reinforcement learning, using an LLM judge's safety score as reward, so the suffix exploits cross-modal information to make the target VLM refuse or respond safely.

Key results

• On BAP attack (MM-SafetyBench), BlueSuffix reduces ASR on LLaVA-v1.5-7B from 61.02% to 4.65%, on MiniGPT-4 from 62.26% to 9.37%, and on InstructionBLIP from 58.48% to 6.78%.
• Against single-modality attacks (VAA, ImgJP, GCG, AutoDAN) BlueSuffix reduces ASR to 0.00% across the open-source VLMs and Gemini-1.5-flash in the reported settings.
• BlueSuffix keeps benign passing rate at 78.00% on LLaVA versus 81.60% without defense (AlpacaEval, GPT-4o judge).

Measured effects reported in the paper

Surprising findings

• Using only the text purifier without the suffix generator can sometimes yield stronger jailbreak prompts, giving higher ASR than no text purifier.
• The suffix generator was trained only on LLaVA responses yet transfers to other target VLMs and to unseen datasets such as RedTeam-2K.

Limitations

• Evaluation of the adaptive attack is limited to one adaptive BAP setting on LLaVA with MM-SafetyBench; authors acknowledge more advanced future attacks may circumvent the defense.
• Safety judgments rely on GPT-4o as judge, and BPR is not 100% for the original prompts due to judging difficulty.
• Gemini results and some baselines are reported only for subsets of settings (dashes in table), limiting comparisons.

Experimental setup

Datasets: MM-SafetyBench AdvBench Harmful_Instructions RedTeam-2K AlpacaEval

Benchmarks: MM-SafetyBench AdvBench Harmful_Instructions RedTeam-2K AlpacaEval

Metrics: attack success rate (ASR) benign passing rate (BPR) mutual information Perspective API toxicity

Baselines: DiffPure Safety Prompt DiffPure + Safety Prompt R2D2 CAT VLGuard

Builds on: DiffPure PPO BAP Jailguard CIDER

Closest papers by meaning

• The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense (NeurIPS 2025 · similarity 0.90)
• MMJ-Bench: A Comprehensive Study on Jailbreak Attacks and Defenses for Vision Language Models (AAAI 2025 · similarity 0.90)
• JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.89)
• Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment (CVPR 2025 · similarity 0.89)
• SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism (NeurIPS 2025 · similarity 0.89)
• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.88)