Research map / Vision-language and multimodal / Multimodal LLMs
Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that MLLM safety gains from curated malicious-data finetuning mainly come from the proportion of rejection responses, not data quality, and that finetuning on benign LLaVA-Med data with simple rejection sentences substantially reduces typographic jailbreak success.
Problem
MLLMs have a safety alignment gap on vision-domain (typographic) jailbreaks, and it is unclear which properties of curated safety datasets (e.g., VLGuard) actually drive the defense.
Approach
Ablations on VLGuard show that image content, reasoning, and multi-turn contrast contribute little; the key factor is the rejection data ratio. Finetuning on benign LLaVA-Med conversations with answers replaced by a fixed rejection sentence (mixed with LLaVA-v1.5-mix665k) recovers much of the safety without curated malicious data.
Key results
- On LLaVA-v1.5-7B, LoRA finetuning with 2000 LLaVA-Med rejection samples reduces MM-SafetyBench ASR from 96.37% to 5.60% and FigStep ASR from 100% to 0.20%, close to VLGuard's 0.18%/0.00%.
- XSTest compliance on LLaVA-v1.5-7B is 90.40% for the proposed method versus 77.20% for VLGuard, while rejection rate is 82.00% versus 96.50%.
- In the rejection-proportion ablation on LLaVA-v1.5-7B, FigStep ASR drops from 100% to 0.20% only with 2000 rejection samples, with ScienceQA and VizWizQA only slightly affected.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on benign data with rejection responses, without curated malicious examples | LLaVA-v1.5 models · MM-SafetyBench and FigStep | ASR: 96.37 -> 5.60 (MM-SafetyBench); 100.00 -> 0.20 (FigStep) | Helped |
| Add rejection data during visual instruction tuning (ablation) | LLaVA-v1.5-7B · MM-SafetyBench | ASR: 96.36 -> 58.93 | Helped |
| Add a small proportion of rejection data during visual instruction tuning (ablation) | LLaVA-v1.5-7B · FigStep | ASR: 100.00 -> 99.40 | No clear effect |
| Increase rejection data to 5% in visual instruction tuning (ablation) | LLaVA-v1.5-7B · MM-SafetyBench and FigStep | ASR: 96.36 -> 60.12 (MM-SafetyBench); 100.00 -> 99.80 (FigStep) | Mixed |
| Split multi-turn safety conversations into single-turn examples (ablation) | VLGuard fine-tuning · MM-SafetyBench | ASR: 0.18 -> 0.18 | No clear effect |
| Remove benign safe-safe conversations (ablation) | VLGuard fine-tuning · MM-SafetyBench | ASR: 0.18 -> 0.18 | No clear effect |
| Replace harmful training images with benign images (ablation) | VLGuard fine-tuning · MM-SafetyBench | ASR: 0.18 -> 0.18 | No clear effect |
| Replace detailed rejection answers with a simple rejection (ablation) | VLGuard fine-tuning · MM-SafetyBench | ASR: 0.18 -> 0.18 | No clear effect |
| Use randomly generated rejection reasons (ablation) | VLGuard fine-tuning · MM-SafetyBench | ASR: 0.18 -> 0.12 | No clear effect |
| Increase the rejection proportion in fine-tuning data (ablation) | MLLMs · XSTest | Compliance: 92.00 -> 52.00 | Hurt |
| Remove ordinary instruction data and train on pure VLGuard (ablation) | LLaVA-v1.5-7B · XSTest | Compliance: 77.20 -> 52.00 | Hurt |
| Use rejection data from LLaVA-v1.5-mix665k rather than LLaVA-Med (ablation) | LLaVA-v1.5-7B · ScienceQA | Total accuracy: 70.12 -> 55.20 | Hurt |
Surprising findings
- Image content, rejection reasons, and multi-turn contrast in the VLGuard dataset contribute little to safety; removing them barely changes defense.
- Raising the rejection proportion causes over-rejection of benign prompts, with models rejecting over 99% of harmful text prompts but answering only about half of benign XSTest questions.
Limitations
- Performance drops on image description queries (rejections on LLaVA-Instruct-150K and MMInstruct).
- The method does not significantly improve text-only jailbreak defense compared to VLGuard.
- Safety evaluation relies on string matching, with Llama-Guard judged to be unreliable for some topics.
Experimental setup
Datasets: LLaVA-v1.5-mix665k LLaVA-Med VLGuard
Benchmarks: MM-SafetyBench FigStep XSTest VizWizQA ScienceQA MMInstruct LLaVA-Instruct-150K
Metrics: attack success rate (string matching) compliance rate rejection rate accuracy
Baselines: VLGuard Original LLaVA-v1.5 Llama-Guard-3-8B evaluation
Builds on: VLGuard LLaVA-v1.5 LoRA
Closest papers by meaning
- GuardAlign: Robust Safety Alignment in Multimodal Large Language Models (ICLR 2026 · similarity 0.92)
- Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (ICML 2024 · similarity 0.92)
- MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.92)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.91)
- Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMs (ICLR 2026 · similarity 0.91)
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.